Audit on Valeri's challenge: our Chandrupatla implementation stops below 2*xtol (tl = tol/span > 1/2) while every other method, SciPy's included, stops at xtol; the protocol corrected that factor for ITP and not here, and best_of let the looser run win. Corrected it costs Chandrupatla 2 evaluations on APS (2785 -> 2787) and 18 on CHA (2423 -> 2441): real, in the competitor's favour, and not what changes the picture. The public scipy.optimize.elementwise.find_root, the private _chandrupatla and ours corrected agree on all 199 problems exactly.

What the tables' totals hid: problem by problem the scheduled projection beats Chandrupatla on APS 87/24/42, exact sign test p = 9e-5, while trailing by 0.7% in total; on CHA Chandrupatla is genuinely better (26 to 9, p = 0.006). The earned projection ties him on APS problems (79/3/71, p = 0.57). Benchmark not yet switched to the library baseline - awaiting Valeri's decision together with the framing of the proposed method. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-21 10:14:42 +01:00
parent 8830765454
commit de90c7a089
6 changed files with 231 additions and 0 deletions
@@ -0,0 +1,62 @@
+"""Audit: is "Chandrupatla is first on the canonical set" a result or a total?
+
+Totals differ by 0.7% (2806 against 2787 with the tolerance corrected); the paper itself
+reads 0.5% as a tie.  Problem by problem, with an exact two-sided sign test on the
+problems where the two differ, for the scheduled projection AND the earned one against
+Chandrupatla at the corrected tolerance.
+
+    python -u audit_head_to_head.py
+"""
+from __future__ import annotations
+
+import math
+
+from audit_chandrupatla_tolerance import ours_half
+from simroot.baselines import best_of, chandrupatla_scipy
+from simroot.pure import pure_root
+from stage12_validation import load, per_problem
+
+
+def sign_test(w, l):
+    n = w + l
+    if n == 0:
+        return 1.0
+    k = min(w, l)
+    return min(1.0, 2.0 * sum(math.comb(n, i) for i in range(k + 1)) / 2.0 ** n)
+
+
+def main():
+    probs = load()
+    ch = per_problem(best_of(ours_half, chandrupatla_scipy), probs, 200)
+    ours = {
+        "scheduled projection (c=2)": lambda f, a, b, xt, mx: pure_root(
+            f, a, b, xt, mx, similarity="ratio", scale="derived3", guard="project"),
+        "earned projection (credit)": lambda f, a, b, xt, mx: pure_root(
+            f, a, b, xt, mx, similarity="ratio", scale="derived3", guard="credit"),
+    }
+    sets = {s: [pid for s_, pid, *_ in probs if s_ == s] for s in ("APS", "CHA")}
+    out = {}
+    for name, m in ours.items():
+        p = per_problem(m, probs, 200)
+        for s, ids in sets.items():
+            w = t = l = 0
+            for pid in ids:
+                (no, oko), (nc, okc) = p[pid], ch[pid]
+                if not (oko and okc):
+                    continue
+                if no < nc:
+                    w += 1
+                elif no > nc:
+                    l += 1
+                else:
+                    t += 1
+            tot_o = sum(p[pid][0] for pid in ids)
+            tot_c = sum(ch[pid][0] for pid in ids)
+            out[(name, s)] = (w, t, l, sign_test(w, l), tot_o, tot_c)
+            print(f"{name:<28} {s}: wins/ties/losses {w}/{t}/{l}   sign test p = "
+                  f"{sign_test(w, l):.2g}   totals {tot_o} vs Chandrupatla {tot_c}")
+    return out
+
+
+if __name__ == "__main__":
+    main()