vvs/sim-numerics
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

PREREG_roots_jl outcome (server): variant A does NOT transfer by the stop rule on the Roots.jl / ModAB benchmark. 78 out-of-sample problems: total 1464 vs Chandrupatla 1566 (-6.5%, R1 holds), per problem 38/10/30 p=0.4 (R2 fails), guarantee held and all solved (R3); strict f24-f91 the same picture (1340 vs 1454, 35/6/25 p=0.25). B does not transfer either (32/11/35). Same structure as the first new corpus: multiple/flat roots 331 vs 481 (10:3); the rest 1133 vs 1085, per problem 28/10/27 - parity. Largest loss f32 (4 - tan x near the pole, 41 vs 12). Against Brent and TOMS748 A wins (47/25 p=0.013, 55/20 p=6e-5); they break the 2x-bisection bound on 11 and 9 problems, A on none.

Browse Source 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
vvs
2026-09-21 19:40:05 +01:00
co-authored by Claude Opus 5
parent 97af3ae767
commit 554ec0c0ae
3 changed files with 228 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results_rootsjl.txt
+123
View File
@@ -0,0 +1,123 @@
== MAIN (78 out-of-sample)
base total 1884 solved 78/78 violations 0
A total 1464 solved 78/78 violations 0
B total 1537 solved 78/78 violations 0
chandrupatla total 1566 solved 78/78 violations 0
brent total 2455 solved 78/78 violations 11
toms748 total 2339 solved 78/78 violations 9
A: R1 ✓ (1464 vs 1566) R2 ✗ (38/10/30 p=0.4) R3 ✓ (viol 0, solved 78 vs 78) -> does NOT transfer
vs brent total 1464 vs 2455 per problem 47/6/25 p=0.013
vs toms748 total 1464 vs 2339 per problem 55/3/20 p=6.5e-05
B: R1 ✓ (1537 vs 1566) R2 ✗ (32/11/35 p=0.81) R3 ✓ (viol 0, solved 78 vs 78) -> does NOT transfer
vs brent total 1537 vs 2455 per problem 43/9/26 p=0.053
vs toms748 total 1537 vs 2339 per problem 46/6/26 p=0.024
== STRICT (f24-f91, out-of-sample)
base total 1734 solved 66/66 violations 0
A total 1340 solved 66/66 violations 0
B total 1385 solved 66/66 violations 0
chandrupatla total 1454 solved 66/66 violations 0
brent total 2333 solved 66/66 violations 11
toms748 total 2216 solved 66/66 violations 9
A: R1 ✓ (1340 vs 1454) R2 ✗ (35/6/25 p=0.25) R3 ✓ (viol 0, solved 66 vs 66)
vs brent total 1340 vs 2333 per problem 41/3/22 p=0.023
vs toms748 total 1340 vs 2216 per problem 48/2/16 p=7.7e-05
B: R1 ✓ (1385 vs 1454) R2 ✗ (29/7/30 p=1) R3 ✓ (viol 0, solved 66 vs 66)
vs brent total 1385 vs 2333 per problem 37/8/21 p=0.048
vs toms748 total 1385 vs 2216 per problem 41/6/19 p=0.0062
== IN-SAMPLE overlaps (13, for reference)
base total 142 solved 13/13 violations 0
A total 153 solved 13/13 violations 0
B total 154 solved 13/13 violations 0
chandrupatla total 135 solved 13/13 violations 0
brent total 128 solved 13/13 violations 0
toms748 total 151 solved 13/13 violations 0
A: R1 ✗ (153 vs 135) R2 ✗ (4/2/7 p=0.55) R3 ✓ (viol 0, solved 13 vs 13)
vs brent total 153 vs 128 per problem 4/1/8 p=0.39
vs toms748 total 153 vs 151 per problem 8/2/3 p=0.23
B: R1 ✗ (154 vs 135) R2 ✗ (3/0/10 p=0.092) R3 ✓ (viol 0, solved 13 vs 13)
vs brent total 154 vs 128 per problem 3/0/10 p=0.092
vs toms748 total 154 vs 151 per problem 5/4/4 p=1
PER PROBLEM, out-of-sample (A / B / Chandrupatla / Brent / TOMS748), * = unsolved:
f01: 8 / 10 / 3 / 10 / 9
f02: 12 / 12 / 10 / 13 / 10
f03: 12 / 42 / 12 / 12 / 17
f04: 9 / 9 / 9 / 10 / 11
f07: 10 / 13 / 11 / 12 / 11
f09: 10 / 9 / 10 / 10 / 11
f10: 10 / 10 / 10 / 9 / 8
f11: 11 / 10 / 10 / 9 / 7
f16: 15 / 10 / 8 / 7 / 9
f21: 10 / 9 / 9 / 10 / 10
f22: 8 / 9 / 10 / 10 / 10
f23: 9 / 9 / 10 / 10 / 10
f24: 26 / 26 / 43 / 122 / 88
f25: 27 / 27 / 43 / 106 / 76
f26: 26 / 26 / 41 / 121 / 95
f27: 17 / 25 / 14 / 32 / 25
f28: 21 / 21 / 47 / 119 / 73
f29: 16 / 16 / 12 / 30 / 25
f30: 17 / 17 / 12 / 13 / 14
f31: 5 / 13 / 13 / 12 / 18
f32: 41 / 11 / 12 / 13 / 14
f33: 17 / 20 / 12 / 14 / 13
f34: 9 / 9 / 10 / 9 / 10
f35: 20 / 22 / 16 / 11 / 15
f36: 49 / 52 / 47 / 39 / 47
f37: 19 / 19 / 17 / 31 / 29
f38: 51 / 48 / 47 / 47 / 51
f39: 59 / 58 / 47 / 73 / 97
f40: 51 / 47 / 47 / 49 / 77
f42: 9 / 9 / 11 / 10 / 10
f43: 25 / 25 / 44 / 106 / 91
f44: 30 / 30 / 44 / 104 / 106
f45: 43 / 43 / 44 / 117 / 123
f46: 14 / 15 / 14 / 15 / 19
f47: 14 / 13 / 15 / 15 / 18
f48: 14 / 19 / 17 / 16 / 15
f49: 13 / 13 / 10 / 11 / 11
f50: 10 / 13 / 10 / 12 / 13
f51: 9 / 9 / 9 / 10 / 10
f52: 18 / 16 / 14 / 17 / 22
f53: 11 / 11 / 12 / 13 / 11
f54: 7 / 7 / 7 / 8 / 9
f55: 8 / 10 / 9 / 11 / 9
f56: 9 / 10 / 9 / 11 / 10
f57: 11 / 11 / 11 / 12 / 16
f58: 8 / 9 / 10 / 10 / 10
f59: 12 / 10 / 9 / 13 / 13
f60: 7 / 7 / 8 / 8 / 8
f61: 10 / 12 / 11 / 11 / 11
f63: 9 / 11 / 10 / 10 / 10
f64: 8 / 8 / 9 / 9 / 9
f65: 8 / 8 / 9 / 9 / 9
f66: 14 / 13 / 12 / 10 / 13
f67: 11 / 11 / 12 / 12 / 12
f68: 7 / 8 / 9 / 8 / 8
f69: 8 / 8 / 10 / 10 / 11
f70: 10 / 10 / 12 / 11 / 13
f71: 5 / 13 / 11 / 13 / 16
f72: 29 / 25 / 48 / 110 / 112
f73: 26 / 26 / 43 / 120 / 91
f74: 26 / 26 / 43 / 122 / 91
f75: 13 / 18 / 27 / 66 / 61
f76: 44 / 44 / 43 / 43 / 43
f77: 51 / 43 / 43 / 46 / 66
f78: 48 / 48 / 43 / 44 / 45
f79: 47 / 43 / 40 / 40 / 39
f80: 23 / 27 / 29 / 21 / 21
f81: 57 / 44 / 50 / 45 / 39
f82: 4 / 4 / 5 / 4 / 5
f83: 32 / 25 / 48 / 106 / 76
f84: 8 / 8 / 13 / 8 / 19
f85: 14 / 53 / 15 / 12 / 13
f86: 19 / 21 / 13 / 16 / 17
f87: 5 / 33 / 11 / 12 / 14
f88: 15 / 15 / 12 / 13 / 13
f89: 16 / 14 / 14 / 14 / 19
f90: 16 / 15 / 12 / 15 / 14
f91: 14 / 14 / 10 / 13 / 15
rootsjl done
sweep_rootsjl.json
+1
View File
File diff suppressed because one or more lines are too long
sweep_rootsjl.py
+104
View File
@@ -0,0 +1,104 @@
"""PREREG_roots_jl_2026-09-21: variants A, B, the plain rule and the three library rivals on
the Roots.jl / ModAB bracketing benchmark (corpus_roots_jl.py).
python -u sweep_rootsjl.py run -> sweep_rootsjl.json, then the verdicts
python -u sweep_rootsjl.py --report verdicts only
"""
from __future__ import annotations
import json
import math
import sys
from multiprocessing import Pool
from corpus_roots_jl import build, solved
from sweep_full import MAXFEV, XTOL, make_method
from sweep_oos import CONFIGS as OOS_CONFIGS
from sweep_oos import sign_p
NAMES = ("base", "A", "B", "chandrupatla", "brent", "toms748")
def evaluate(name):
method = make_method(OOS_CONFIGS[name])
probs, _ = build()
res = {}
for pid, f, a, b, ins in probs:
try:
x, n, conv = method(f, a, b, XTOL, MAXFEV)
x = float(x)
except Exception:
x, n, conv = float("nan"), MAXFEV, False
ok = bool(conv and solved(f, x))
over = ok and int(n) > 2 * math.ceil(math.log2(abs(b - a) / XTOL)) + 4
res[pid] = [int(n), ok, bool(over), bool(ins)]
return name, res
def cost(v):
return v[0] if v[1] else MAXFEV
def report(R):
keep_main = lambda pid, v: not v[3]
keep_strict = lambda pid, v: not v[3] and int(pid[1:]) >= 24
keep_ins = lambda pid, v: v[3]
ch = R["chandrupatla"]
for title, keep in (("MAIN (78 out-of-sample)", keep_main),
("STRICT (f24-f91, out-of-sample)", keep_strict),
("IN-SAMPLE overlaps (13, for reference)", keep_ins)):
print(f"\n== {title}")
for k in NAMES:
r = R[k]
sel = {p: v for p, v in r.items() if keep(p, v)}
print(f" {k:<14} total {sum(cost(v) for v in sel.values()):6d} "
f"solved {sum(v[1] for v in sel.values()):3d}/{len(sel)} "
f"violations {sum(v[2] for v in sel.values())}")
for v_name in ("A", "B"):
r = R[v_name]
sel = [p for p, v in r.items() if keep(p, v)]
tot = sum(cost(r[p]) for p in sel)
tot_ch = sum(cost(ch[p]) for p in sel)
w = sum(cost(r[p]) < cost(ch[p]) for p in sel)
l = sum(cost(r[p]) > cost(ch[p]) for p in sel)
t = len(sel) - w - l
viol = sum(r[p][2] for p in sel)
sv, sc = sum(r[p][1] for p in sel), sum(ch[p][1] for p in sel)
r1, r2, r3 = tot < tot_ch, w > l and sign_p(w, l) < 0.05, viol == 0 and sv >= sc
line = (f" {v_name}: R1 {'✓' if r1 else '✗'} ({tot} vs {tot_ch}) R2 {'✓' if r2 else '✗'} "
f"({w}/{t}/{l} p={sign_p(w, l):.2g}) R3 {'✓' if r3 else '✗'} (viol {viol}, "
f"solved {sv} vs {sc})")
if keep is keep_main:
line += " -> " + ("TRANSFERS" if r1 and r2 and r3 else "does NOT transfer")
print(line)
for rn in ("brent", "toms748"):
o = R[rn]
w2 = sum(cost(r[p]) < cost(o[p]) for p in sel)
l2 = sum(cost(r[p]) > cost(o[p]) for p in sel)
print(f" vs {rn:<8} total {tot} vs {sum(cost(o[p]) for p in sel)} "
f"per problem {w2}/{len(sel) - w2 - l2}/{l2} p={sign_p(w2, l2):.2g}")
print("\nPER PROBLEM, out-of-sample (A / B / Chandrupatla / Brent / TOMS748), * = unsolved:")
for p, v in R["A"].items():
if v[3]:
continue
cells = []
for k in ("A", "B", "chandrupatla", "brent", "toms748"):
c = R[k][p]
cells.append(f"{c[0]}{'' if c[1] else '*'}")
print(f" {p}: " + " / ".join(cells))
def main():
if "--report" not in sys.argv:
with Pool(len(NAMES)) as pool:
R = dict(pool.map(evaluate, NAMES))
with open("sweep_rootsjl.json", "w", encoding="utf8") as fh:
json.dump(R, fh)
with open("sweep_rootsjl.json", encoding="utf8") as fh:
R = json.load(fh)
report(R)
print("rootsjl done")
if __name__ == "__main__":
main()