Separate review and second candidate

GeoStep 0.4.0rc2 addresses the separate review of the first release candidate. The numerical core supports a limited candidate release. The experimental-method restrictions remain; this is not approval of general nominal calibration.

Review independence and scope

A separate AI reviewer inspected the analysers, inference, allocation, historical power, frozen protocol and retained evidence. The reviewer did not implement the original repairs or change files, packages or Git state. This is a second-agent review, not an external human statistical sign-off.

The reviewer independently recomputed every retained cell’s bias, Monte Carlo standard error, RMSE, coverage, rejection rate, Wilson interval and mean interval width. All 48 cells and 12,000 records agreed. They also regenerated one panel in each of twelve scenario/effect cells and reproduced its four public analyses, including inputs, truth, seed and numerical outputs. These checks audited the first candidate’s evidence before the numerical correction below.

Findings and corrections

Finding

Consequence

Correction

Absolute permutation comparison tolerance

Changing outcome units could change an exact p-value

Centre outcomes within strata, use the same statistic for observed and permuted allocations, and use a relative floating-point tolerance

Custom significance helper ignored failed status

A failed result could be reported significant

Require success and validate the supplied alpha

CLI dropped a named CRT stratification column

Unsupported stratified CRT could bypass its rejection gate

Pass the declared column through to the analyser

The first finding was confirmed independently by the reviewer. Treatment values [8, 9, 10, 11] and control values [1, 2, 3, 4] have exact two-sided p = 2/70. Scaling every value by 1e-14 previously changed the p-value to one. The repaired calculation retains 2/70. Regression checks cover small and large scales, large representable offsets and seeded Monte Carlo sampling.

The SciPy permutation-test documentation explains floating-point tie handling. GeoStep scales the tolerance to the weighted centred-outcome magnitude. A tolerance based only on the observed contrast missed a complement tie when that contrast was near zero. Exact rational arithmetic confirmed the correction on a retained simulation case. GeoStep retains its absolute-statistic two-sided tail convention; it does not adopt SciPy’s twice-the-smaller-tail definition for asymmetric distributions.

The two result/CLI boundary defects were found by the implementation agent during the separate review. Their regression tests failed before the fixes and pass after them. Valid-result point estimates, variance calculations, confidence intervals, allocation probabilities and inference restrictions are preserved.

The reviewer rechecked the permutation repair, including unequal arm sizes, and reported no remaining consequential methodological blocker in the reviewed scope. Their initial post-repair checks were nine numerical-reference tests and four exact-randomisation regressions. After the near-zero cancellation fix, the reviewer compared every allocation in two strata of four and five geos at three scales against exact rational arithmetic. All 180 comparisons matched. Five focused randomisation regressions also passed. The implementation agent owns the broader verification and calibration refresh below.

Evidence continuity

The first candidate’s summary, replicate records and frozen protocol are preserved. Its package files remain in dist/0.4.0rc1/. The first candidate’s verification record remains historical evidence.

Protocol v2 changes the candidate identifier only. Data generation, seeds, repetitions, resample counts and acceptance rules are unchanged. The current study and its JSON records refer to the second candidate after rerunning against the corrected source. The repeated simulations are a regression check using the same panels, not another independent sample.

Executed checks for the second candidate

  • Development environment, Python 3.12.11: 184 tests passed, seven warnings.

  • Clean installed wheel, Python 3.11.15 with declared minimum dependencies: 184 tests passed, 21 warnings.

  • Clean installed wheel, Python 3.12.11 with resolved permitted dependencies: 184 tests passed, seven warnings.

  • Dependency consistency checks passed in both clean environments. Versions are retained for Python 3.11 and Python 3.12.

  • The retained-allocation example ran from the installed second candidate and reproduced its estimate and interval after reloading saved files.

  • Ruff and Black passed on inference.py, run_pipeline.py and tests/test_review_followup.py. The schema edit was exercised by regression tests; whole-tree lint or formatting cleanliness is not claimed.

  • Documentation built with Sphinx -W --keep-going, without suppressing broken reference warnings. The site was not deployed.

  • A third clean Python 3.12 environment installed from the source archive: 184 tests passed, seven warnings. Dependency consistency and the retained allocation example passed.

  • Both distribution formats passed twine check. Archive inspection confirmed the licence, exact package-source parity with all tested installations, inclusion of both candidates’ evidence and exclusion of stale generated HTML.

  • Final files are in dist/0.4.0rc2/, with a SHA256 manifest. The first candidate’s files remain unchanged. Documentation-only rebuilds preserve the tested source.

Warnings remain constant-data diagnostics and third-party deprecations. No blanket warning suppression was added. Tests run outside the checkout use a copy of tests, scripts, examples and the runner, with no source directory and no PYTHONPATH override.

The protocol-v2 rerun completed 12,000 analysis calls with zero failures. All cell summaries and numerical replicate records match rc1 exactly. Final source and protocol hashes match the retained JSON. Scale and cancellation regressions provide essential evidence alongside this simulation grid.

Release decision

Retain BCa, stratified DiD, CRT and staircase inference as experimental. Do not promote the package to a general stable-calibration claim. A 90% coverage floor and 10% rejection ceiling remain screening tolerances; they do not certify nominal 95% intervals or 5% tests. The small-sample and causal-design limits still apply.

No commit, push, publication or deployment is part of this review follow-up.