Comparative table of wins and losses (server numbers): five variants of the proposed rule against library Chandrupatla, Brent, TOMS748 - totals, per-problem sign tests, wide-bracket families. Server defcontra grid matches the smoke in every projection cell but one (defect+G2 CHA 3183 vs 3182).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-21 16:00:05 +01:00
parent 3d9287ec2c
commit 63cf9fb768
3 changed files with 91 additions and 0 deletions
@@ -0,0 +1,33 @@
+TOTALS (evaluations of f, failures charged 200)
+  method                              APS    CHA   both
+  proposed rule                      2806   3220   6026
+  + G2 (step pattern)                2782   3169   5951
+  + G2 + analogy entry               2780   2625   5405
+  + contra + remembered analogy      2792   2537   5329
+  + defect + remembered analogy      2530   2561   5091
+  Chandrupatla                       2787   2441   5228
+  Brent                              2889   4115   7004
+  TOMS748                            2820   4000   6820
+
+PER PROBLEM, ours against each rival: wins/ties/losses (sign-test p)
+  APS
+    proposed rule                    Chandrupatla 87/24/42 p=9.2e-05   Brent 102/11/40 p=2e-07   TOMS748 86/16/51 p=0.0035
+    + G2 (step pattern)              Chandrupatla 95/20/38 p=8.2e-07   Brent 110/9/34 p=1.5e-10   TOMS748 97/15/41 p=2.1e-06
+    + G2 + analogy entry             Chandrupatla 95/20/38 p=8.2e-07   Brent 110/9/34 p=1.5e-10   TOMS748 97/15/41 p=2.1e-06
+    + contra + remembered analogy    Chandrupatla 90/23/40 p=1.4e-05   Brent 105/11/37 p=9.9e-09   TOMS748 87/16/50 p=0.002
+    + defect + remembered analogy    Chandrupatla 86/20/47 p=0.00091   Brent 101/13/39 p=1.6e-07   TOMS748 107/11/35 p=1.1e-09
+  CHA
+    proposed rule                    Chandrupatla 9/5/26 p=0.006   Brent 30/4/4 p=6.2e-06   TOMS748 31/1/8 p=0.00029
+    + G2 (step pattern)              Chandrupatla 10/3/27 p=0.0076   Brent 30/2/6 p=7e-05   TOMS748 31/1/8 p=0.00029
+    + G2 + analogy entry             Chandrupatla 9/6/25 p=0.009   Brent 30/2/6 p=7e-05   TOMS748 36/1/3 p=3.6e-08
+    + contra + remembered analogy    Chandrupatla 8/7/25 p=0.0046   Brent 31/4/3 p=7.7e-07   TOMS748 37/1/2 p=2.8e-09
+    + defect + remembered analogy    Chandrupatla 8/5/27 p=0.0019   Brent 30/2/6 p=7e-05   TOMS748 35/1/4 p=3.4e-07
+
+CHA PER FAMILY (ours / Chandrupatla)
+                                       fun1     fun2     fun3     fun4     fun5     fun6     fun7     fun8     fun9
+  proposed rule                          99      141      338      573      456      456     1000       52      105
+  + G2 (step pattern)                    97      146      346      522      456      456     1000       52       94
+  + G2 + analogy entry                   99      146      336      331      254      313     1000       52       94
+  + contra + remembered analogy         108      141      304      318      254      275     1000       52       85
+  + defect + remembered analogy         123      144      304      315      257      280     1000       57       81
+  Chandrupatla                          103      152      305      297      233      233     1000       63       55
@@ -0,0 +1,57 @@
+"""Comparative table of our wins and losses (server results only).
+
+Variants of the proposed rule (tent, 3-point, projection) against the library Chandrupatla,
+Brent (brentq) and TOMS748: totals per set, both sets, and per-problem wins/ties/losses with
+the sign test; then the wide-bracket set per family.
+
+    python -u table_wins_losses.py
+"""
+from __future__ import annotations
+
+import json
+
+from analyze_extrap import h2h, sign_p
+from where_we_lose import per_family
+
+
+def load(p):
+    with open(p, encoding="utf8") as fh:
+        return {r["label"]: r for r in json.load(fh)}
+
+
+def main():
+    full = load("sweep_full.json")
+    dc = load("sweep_defcontra.json")
+    rivals = {"Chandrupatla": full["chandrupatla"], "Brent": full["brent"],
+              "TOMS748": full["toms748"]}
+    ours = {
+        "proposed rule": full["pure.ratio.derived3.project"],
+        "+ G2 (step pattern)": dc["pure+extP.ratio.derived3.project"],
+        "+ G2 + analogy entry": dc["pure+extP+flatanalog.ratio.derived3.project"],
+        "+ contra + remembered analogy": dc["contra+flatanalogmem.ratio.derived3.project"],
+        "+ defect + remembered analogy": dc["defect+flatanalogmem.ratio.derived3.project"],
+    }
+    print("TOTALS (evaluations of f, failures charged 200)")
+    print(f"  {'method':<32} {'APS':>6} {'CHA':>6} {'both':>6}")
+    for name, r in list(ours.items()) + list(rivals.items()):
+        print(f"  {name:<32} {r['APS']['total']:6d} {r['CHA']['total']:6d} "
+              f"{r['APS']['total'] + r['CHA']['total']:6d}")
+    print("\nPER PROBLEM, ours against each rival: wins/ties/losses (sign-test p)")
+    for s in ("APS", "CHA"):
+        print(f"  {s}")
+        for name, r in ours.items():
+            cells = []
+            for rn, rv in rivals.items():
+                w, t, l = h2h(r, rv, s)
+                cells.append(f"{rn} {w}/{t}/{l} p={sign_p(w, l):.2g}")
+            print(f"    {name:<32} " + "   ".join(cells))
+    print("\nCHA PER FAMILY (ours / Chandrupatla)")
+    fams = sorted(per_family(rivals["Chandrupatla"], "CHA"))
+    print("  " + " " * 32 + "".join(f"{f:>9}" for f in fams))
+    for name, r in list(ours.items()) + [("Chandrupatla", rivals["Chandrupatla"])]:
+        pf = per_family(r, "CHA")
+        print(f"  {name:<32}" + "".join(f"{pf[f]:9d}" for f in fams))
+
+
+if __name__ == "__main__":
+    main()