Test coverage gaps – what to add and harden#
Part of WP0 (WP0_Test_Baseline.md) of the migration plan. This compares the commands
documented in the help files with what the current macro tests actually run
and check. It then lists the tests to add or strengthen, ordered by when the
migration work packages (WP3–WP9) need them. mixscat is deprecated and not
included.
Generated from the run of 2026-11-13 (claude-python-app after PR #33; 31
tests). Re-run at any time:
python3 tools/run_legacy_tests.py --exe /path/to/discus_suite -j 4 --workdir /tmp/run
python3 tools/command_coverage.py --logs /tmp/run --markdown coverage.md
1. Summary#
Program |
Commands reached |
Sub-commands / options reached |
Comment |
|---|---|---|---|
DISCUS |
28 / 61 |
136 / 402 |
Core scattering paths covered; I/O, symmetry, many structure-editing commands, RMC and Patterson not reached |
KUPLOT |
34 / 82 |
9 / 69 |
Used mainly as a test harness ( |
DIFFEV |
0 / 28 |
– |
Not tested at all |
REFINE |
6 / 9 |
3 / 4 |
One test |
SUITE |
3 / 7 |
– |
|
Command language ( |
19 / 31 |
– |
Covered as a side effect; |
EXPERI |
0 / 1 |
0 / 10 |
|
Reached is not the same as checked. Many tests run a command but only check something simple afterwards. Section 3 rates each test’s check.
How the numbers are made, and their limits:
Commands come from the help files: level 2 is a command, level 3 is a sub-command or option.
What ran comes from the command echo in the test logs, matched by unique prefix, since the command language accepts abbreviations.
Macro echo also prints the lines of
ifbranches that are not taken. Commands insidedoloops in a sub-menu are echoed with the outer prompt. A few counts can therefore be off; for example,transformcounts as reached because atrans …line inside a menu loop matched it.Options passed as parameters (for example
set technique:nufft) are only counted forset. Everything else is counted at command level.
2. Gaps by migration work package#
WP3 – Crystal model and I/O (needed first)#
Gap |
Today |
Why it matters |
|---|---|---|
Structure export/import: |
Covered ( |
The Python |
|
|
Round trip Python → Fortran → file |
|
Space-group expansion covered: 10 structures (C2/c, Cmcm, Cmc2₁, Cmm2, F-43m, Pn-3m, Fm-3m, P4mm, P6₃/mmc, I4₁/amd CIF) agree site by site with gemmi; supercells checked. |
Space-group expansion is wrapped in WP3 |
|
No sub-commands reached |
Cell transformations are part of the data model |
|
Mostly not reached |
Small metric helpers; cheap exact tests |
KUPLOT |
3 / 16 formats |
Data-file parity when I/O moves to h5py |
WP4 – Single-crystal Fourier#
Gap |
Today |
Why it matters |
|---|---|---|
Radiation and energy: |
Covered ( |
Every radiation type needs a golden result |
Anomalous dispersion |
|
Options the API must expose |
|
NUFFT is wrong for small crystals on many grids (e.g. Ni 4×4×4, hk0 −3…3: R = 1.7, spurious maxima every 0.08–0.16 in h); exact for the 50×50 layer and some size/grid combinations. Known-failure test; the default technique is unaffected. Likely related to the rare broken Fourier result in |
The two methods should agree: a cheap consistency test |
Only |
|
Check the other |
|
Not reached |
|
Strong checks exist (R < 0.001 against |
02.1d, 02.1d_periodic, 02.2d |
Good model for the rest |
WP5 – Powder#
Gap |
Today |
Why it matters |
|---|---|---|
Neutron and electron powder ( |
Not reached |
|
One parameter set only |
All 3 powder runs use the same |
Need variations: Debye vs. complete vs. FFT method, profile shapes, preferred orientation, 2θ vs. Q output |
WP6 – PDF, 3D-PDF, Patterson#
Gap |
Today |
Why it matters |
|---|---|---|
|
Runs 9 PDF calculations; nothing is compared |
Highest priority: add a G(r) comparison against a reference |
PDF fitting mode ( |
Not reached |
|
|
Not reached |
Wrapped in WP6 |
3D-PDF |
|
Easy win: compare them in the SRO tests |
|
Not reached |
Clarify whether EXPERI is still used |
WP7 – Structure building and defects#
Gap |
Today |
Why it matters |
|---|---|---|
Atom editing: |
Not reached |
Small deterministic tests with exact atom-position checks |
|
Only the plain form |
|
Molecules: |
Not reached; |
|
|
Not reached |
|
|
Not reached |
|
|
15 / 26 |
|
|
7 / 18 |
Analysis output is exact and easy to compare |
|
Reached, no sub-commands |
Used by decoration and molecules |
WP8 – Monte Carlo#
Gap |
Today |
Why it matters |
|---|---|---|
|
Not reached |
Wrapped in WP8 |
|
5 tests, all with statistical tolerances (R < 0.25–0.6, correlation ±0.04–0.09) |
Needs |
|
Never used in a test |
Fix first (0.5a), then use it everywhere |
WP9 – Refinement and plotting#
Gap |
Today |
Why it matters |
|---|---|---|
DIFFEV: 0 / 28 commands |
Not tested. Candidates exist in |
The Python DE replacement needs reference behaviour |
REFINE |
Not reached |
|
KUPLOT |
Not reached |
Compare against lmfit/scipy in WP9 |
KUPLOT data processing: |
Mostly not reached |
Decide: if these move to NumPy, test only what users rely on |
KUPLOT plot styling (colours, fonts, frames, titles) |
Reached but not checked |
Do not test. Replaced by matplotlib/plotly |
MPI runs ( |
Disabled |
Nightly or manual job |
Command language and suite#
The command language is being retired, so test only what macros still depend
on. if, do, variable, fopen, fput and similar are already covered as
a side effect. Exceptions:
seed: needed for reproducible tests (0.5a).Error handling: no test checks that a bad command gives the expected error. WP2 maps errors to Python exceptions, so a small set of “this input must fail with error N” tests is worth having.
3. Strength of the existing checks#
Check |
Tests |
Assessment |
|---|---|---|
Exact reference data, tight tolerance (R < 0.001–0.01 against |
02.1d, 02.1d_periodic, 02.2d, 02.powder, 08.nano_simple, 08.nano_ellipsoid, 12.waves_density, 13.analyze |
Strong. Model for new tests |
Statistical / loose |
02.2d_lots (R < 0.13–0.15), 04.sro0/1/3/4 (R < 0.25–0.6), 05.stack_growth, 06.perovskite, 07.refine, 06.triangle (intensity ratios) |
Catches large breaks only. Tighten after |
Structural only (atom counts, a few positions, cell size) |
01.circle, 01.checkerboard, 01.symmetry, 01.shear, 05.stack, 05.stack_file, 05.stack_types, 06.cube, 06.tilt, 11.properties, 12.waves (internal consistency), 14.deco01_all, 14.deco02 |
Checks the structure is plausible, not that it is right. Add golden structure files (WP0 task 0.6) |
None |
03.pdf |
Add a check (task 0.4) |
Plot macros now write CIF files (program cif), but nothing compares them.
08.Nanoparticle_example/SIMPLE/PLOT/block.cif came out byte-identical to the
committed copy, so these could become cheap golden checks.
4. Work list#
4.1 Repair existing tests that are not in the suite#
These macros check themselves, but test_all.mac does not run them. Run
through the new runner, none of them passes today:
Macro |
Result |
Likely cause |
|---|---|---|
|
6 FAIL, 3 PASS (two of the passes are false) |
Re-checked: the CIF, VASP and POSCAR differences are format changes (version string, newer CIF symmetry block), not wrong atoms; the CIF round trip is exact (Python tests). The SHELX export is wrong (whole cell written with the space-group symmetry). The RMCProfile and SCATTY comparisons name files that do not exist ( |
|
– |
Uses the legacy atom format (bare |
|
FAIL |
Commented out in |
|
ABORTED: error opening a file |
Input file missing or renamed |
|
ABORTED: KUPLOT “Incompatible data sets for KCAL” |
Output format changed since the reference was made |
4.2 Harden existing tests#
# |
Task |
WP0 task |
|---|---|---|
H1 |
|
0.4 |
H2 |
SRO tests: also compare the existing |
0.4 / 0.6 |
H3 |
Store golden outputs for the “structural only” tests (final structure, CIF) and compare atom positions |
0.6 |
H4 |
After |
0.5a / 0.5 |
H5 |
Compare the CIF files written by the plot macros |
0.6 |
H6 |
Run deterministic tests with 1 and N OpenMP threads and compare |
0.5 |
4.3 New tests, by priority#
Ordered by when the migration needs them. Each test should be small (under 10 s), deterministic (fixed seed), compare one feature against a golden result, and need no interactive viewer.
ID |
Area |
Test |
Check |
Needed by |
|---|---|---|---|---|
A1 |
I/O |
|
exact atoms |
WP3 |
A2 |
I/O |
|
file + atoms |
WP3 |
A3 |
Symmetry |
|
atom count + positions |
WP3 |
A4 |
Metric |
|
exact numbers |
WP3 |
A5 |
Fourier |
X-ray / neutron / electron, |
golden intensity |
WP4 |
A6 |
Fourier |
Standard vs NUFFT technique on the same structure |
agree within tolerance |
WP4 |
A7 |
Fourier |
3D volume and lots variants |
golden |
WP4 |
A8 |
Powder |
Neutron and electron; Debye vs complete vs FFT; profiles; preferred orientation; 2θ vs Q |
golden pattern |
WP5 |
A9 |
X-ray and neutron G(r) with damping/broadening options |
golden G(r) |
WP6 |
|
A10 |
Patterson / 3D-PDF |
|
golden map |
WP6 |
A11 |
exp2pdf |
DISCUS |
golden G(r) |
WP6 |
B1 |
Editing |
|
exact atoms |
WP7 |
B2 |
Insert |
|
exact atoms |
WP7 |
B3 |
Molecules |
|
molecule table |
WP7 |
B4 |
Defects |
|
exact / golden |
WP7 |
B5 |
Stack |
|
golden |
WP7 |
B6 |
Analysis |
|
exact numbers |
WP7 |
B7 |
Waves |
remaining wave types ( |
golden |
WP7 |
C1 |
MC |
|
exact |
WP8 |
C2 |
MC |
|
R-value + seed |
WP8 |
C3 |
MC |
old |
WP8 |
|
D1 |
DIFFEV |
small DE fit (polynomial or |
final parameters |
WP9 |
D2 |
REFINE |
|
parameters + errors |
WP9 |
D3 |
KUPLOT |
|
parameters |
WP9 |
D4 |
KUPLOT |
data processing actually used by users (after the keep/drop decision) |
golden arrays |
WP9 |
E1 |
Errors |
Bad commands, bad parameters and missing files give the expected error number; the run continues |
error code |
WP2 |
E2 |
Command language |
|
exact |
0.5a |
4.4 Out of scope (decide once, then document)#
KUPLOT plot styling (colours, fonts, frames, titles, marks): replaced by matplotlib/plotly, so not worth testing.
Command-language features not used by any macro (
learn,mount,fsub,timer, sockets): retire rather than test.mixscat: deprecated.
EXPERI: test only if it is still used.
5. Conventions for new tests#
One feature per macro, small crystal, run time under 10 s.
seedat the start of every stochastic test (once 0.5a is fixed).Compare against stored reference data (
RESULTS/now,golden/from task 0.6), not only against counts or ranges.Plot macros use
program cifandrun, neverprogram jmolorrun plot:inter.Add every test to
test_all.macandmanifest.tomltogether. Tag it (fast,mc, …), and check withtools/run_legacy_tests.py --check-test-all.After adding tests, re-run
tools/command_coverage.pyand update the appendix below.
Appendix – command coverage tables (generated)#
Output of tools/command_coverage.py for the run above.
Program |
Commands reached |
Sub-commands / options reached |
|---|---|---|
suite |
3 / 7 |
– |
discus |
28 / 61 |
136 / 402 |
kuplot |
34 / 82 |
9 / 69 |
diffev |
0 / 28 |
– |
refine |
6 / 9 |
3 / 4 |
experi |
0 / 1 |
0 / 10 |
lib |
19 / 31 |
– |
suite#
Not reached: diffev, parallel, show, update
Command |
Tests |
Sub-commands reached |
Sub-commands not reached |
|---|---|---|---|
|
30 |
– |
– |
|
6 |
– |
– |
|
1 |
– |
– |
discus#
Not reached: anis, append, asym, bragg, change, copy, d2r, define, demolecularize, diff-four, displacement, exp2pdf, export, import, index2cell, inverse, kick, mc, molecularize, patterson, perioditize, private, project, r2d, remove, rmc, seed, spacegroup, storage, superspace, switch, ther, wyckoff
Command |
Tests |
Sub-commands reached |
Sub-commands not reached |
|---|---|---|---|
|
1 |
– |
– |
|
25 |
– |
– |
|
16 |
7 / 18 |
angle, apply_symmetry, bval, coordination, disp, env, field, homo, mode, neig, trans |
|
1 |
0 / 9 |
add, create, delete, recreate, remove, reset, set, show, switch |
|
2 |
5 / 5 |
– |
|
26 |
7 / 8 |
instructions |
|
2 |
– |
– |
|
13 |
21 / 27 |
calc, delf, electron, energy, hkl, layer |
|
1 |
0 / 14 |
adp, cent, character, density, file, fuzzy, origin, reset, run, show, type, xaxis, yaxis, zaxis |
|
5 |
4 / 5 |
apply_symmetry |
|
16 |
4 / 9 |
input, range, reset, show, threshold |
|
1 |
5 / 9 |
data, i/jdese, i/jsele, run |
|
26 |
16 / 28 |
asym, back, bonds, chem, col, hkl, line, mdes, mole, msel, title, type |
|
3 |
5 / 7 |
electron, neut |
|
3 |
1 / 2 |
set/clear |
|
6 |
– |
– |
|
30 |
4 / 5 |
exit |
|
11 |
– |
– |
|
4 |
– |
– |
|
13 |
4 / 13 |
asym, chem, des, format, incl, omit, prop, reset, show |
|
2 |
6 / 20 |
calc, des, eigen, hkl, mdes, mincl, mode, msel, odes, oincl, osel, show, uvw, vector |
|
11 |
5 / 18 |
adp, asym, bval, chem, config, connectivity, envi, error, mass, menv, object, scat, symmetry |
|
4 |
11 / 19 |
ccol, cele, dist, four, random, reset, rotate, sigma |
|
5 |
2 / 4 |
character, show |
|
3 |
14 / 24 |
asym, calc, chem, des, hkl, mdes, odes, oincl, osel, use |
|
1 |
0 / 33 |
anew, aold, asnew, asold, asym, bnew, bold, bsnew, bsold, c2new, c2old, chem, cnew, cold, csnew, csold, des, h2new, h2old, incl, onew, oold, reset, run, sel, set, show, xnew, xold, ynew, yold, znew, zold |
|
1 |
– |
– |
|
2 |
15 / 26 |
asym, chem, dens, mdes, mrep, msel, opti, osci, rot, shift, tran |
kuplot#
Not reached: adt, alloc, bond, branch, close, color, convolute, cost, derivative, dsave, ecolor, etype, excl, fft, fill, fit, fourier, function, glat, grid, hcolor, hlab, hpak, htype, ident, integral, kpar_par, kpara, lart, mean, merge, mouse, nexus, print, ptype, rdef, rebin, sann, scale, sdef, show, sleg, sm, smax, sort, spline, torefine, window
Command |
Tests |
Sub-commands reached |
Sub-commands not reached |
|---|---|---|---|
|
23 |
– |
– |
|
23 |
– |
– |
|
9 |
– |
– |
|
10 |
– |
– |
|
19 |
– |
– |
|
12 |
– |
– |
|
1 |
– |
– |
|
9 |
– |
– |
|
22 |
– |
– |
|
5 |
1 / 4 |
color, just, typ |
|
12 |
4 / 7 |
bframe, cframe, frame |
|
12 |
– |
– |
|
9 |
– |
– |
|
9 |
– |
– |
|
1 |
– |
– |
|
1 |
1 / 6 |
form, m999, run, show, threshold |
|
3 |
– |
– |
|
28 |
3 / 16 |
csv, de, gs, h5, ma, mc, mp, mrc, nexus, pg, sc/st, xx, zz |
|
2 |
– |
– |
|
1 |
– |
– |
|
24 |
– |
– |
|
2 |
– |
– |
|
1 |
– |
– |
|
19 |
– |
– |
|
2 |
– |
– |
|
3 |
– |
– |
|
26 |
– |
– |
|
26 |
– |
– |
|
16 |
– |
– |
|
3 |
– |
– |
|
24 |
– |
– |
|
1 |
– |
– |
|
22 |
– |
– |
|
22 |
– |
– |
diffev#
Not reached: adapt, allocate, backup, branch, compare, constraint, data, deallocate, dismiss, donor, fix, initialize, lastfile, logfile, newparam, purge, read, refine, release, reset, restart, restrial, run_mpi, selection, show, trialfile, type, write
refine#
Not reached: finished, fix, sigma
Command |
Tests |
Sub-commands reached |
Sub-commands not reached |
|---|---|---|---|
|
1 |
– |
– |
|
1 |
– |
– |
|
1 |
– |
– |
|
1 |
– |
– |
|
1 |
3 / 4 |
log |
|
1 |
– |
– |
experi#
Not reached: exp2pdf
lib#
Not reached: cd, fend, fexist, fsub, input, learn, lend, mount, seed, sleep, timer, umount
Command |
Tests |
Sub-commands reached |
Sub-commands not reached |
|---|---|---|---|
|
1 |
– |
– |
|
10 |
– |
– |
|
24 |
– |
– |
|
30 |
– |
– |
|
16 |
– |
– |
|
31 |
– |
– |
|
31 |
– |
– |
|
5 |
– |
– |
|
1 |
– |
– |
|
31 |
– |
– |
|
31 |
– |
– |
|
30 |
– |
– |
|
1 |
– |
– |
|
31 |
– |
– |
|
1 |
– |
– |
|
30 |
– |
– |
|
30 |
– |
– |
|
31 |
– |
– |
|
19 |
– |
– |