PREREG_out_of_sample outcome (server). Variant A (G2 + contra + remembered analogy + order point + exp limit + half) TRANSFERS: new corpus 3492 vs Chandrupatla 3899, per problem 105/34/67 p=0.0046, guarantee held, solved 205 = 205. Variant B (defect + remembered + point + exp) does NOT (total 3587 below, but per problem 95/33/78 p=0.22; loses the exponential class 9/5/26). Per class the transfer is carried by the multiple/flat-root class C3 (1194 vs 1792, 37/0/3); on C0-C2 A is at par per problem and ~6% dearer in total, C4 ~ par, C5 (near-cancelling) 691 vs 574. Ablation: every component earns its place on at least one set (contra only on APS/CHA: +9/+22, NEW -8; G2 only on NEW: +82). Addendum: C0.05 is SciPy's random step function - seeded and its cache cleared before every run for reproducibility; two unseeded runs gave the same verdicts.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude Opus 5
@@ -49,3 +49,12 @@ C5 почти сокращающиеся члены (как у fun9, парам
|
||||
- B: `defect`; `exp`; точка; весь плоский режим.
|
||||
|
||||
Составляющая с Δ ≤ 0 на всех трёх наборах **не заслужена** и из предложения убирается.
|
||||
|
||||
## Дополнение (после двух прогонов, до сервера) — воспроизводимость
|
||||
|
||||
Задача C0.05 = SciPy `original.05.00` — **случайная ступенчатая функция** (`random()` без зерна,
|
||||
значения в общем кэше модуля, разделяемом всеми прогонами процесса). Два локальных прогона без
|
||||
зерна разошлись только на ней (на 2–9 вызовов у каждого метода, включая соперников) и дали
|
||||
**одинаковые вердикты** (A: O1–O3 ✓; B: O2 ✗). Поправка: перед каждым прогоном кэш очищается, зерно
|
||||
20260921 — все методы встречают одну и ту же функцию, прогон воспроизводим. Задача остаётся в корпусе;
|
||||
критерии и остальной корпус не меняются.
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
TOTALS APS CHA NEW (NEW C0-C2 NEW C3-C5)
|
||||
base 2806 3220 5572 ( 1208 4364)
|
||||
A 2559 2047 3492 ( 1133 2359)
|
||||
B 2531 1972 3587 ( 1150 2437)
|
||||
chandrupatla 2787 2441 3899 ( 1067 2832)
|
||||
brent 2889 4115 6576 ( 1196 5380)
|
||||
toms748 2820 4000 5914 ( 1251 4663)
|
||||
|
||||
OUT OF SAMPLE (NEW) - verdicts
|
||||
A: O1 ✓ (3492 vs 3899) O2 ✓ (105/34/67 p=0.0046) O3 ✓ (viol 0, solved 205 vs 205) O4 ✓ -> TRANSFERS
|
||||
vs brent total 3492 vs 6576 per problem 139/34/33 p=1.1e-16
|
||||
vs toms748 total 3492 vs 5914 per problem 159/27/20 p=4.6e-28
|
||||
per class (ours / Chandrupatla, per problem w/t/l):
|
||||
C0: 173 / 156 2/1/3 p=1
|
||||
C1: 524 / 474 18/5/17 p=1
|
||||
C2: 436 / 437 16/14/10 p=0.33
|
||||
C3: 1194 / 1792 37/0/3 p=1.9e-08
|
||||
C4: 474 / 466 16/5/19 p=0.74
|
||||
C5: 691 / 574 16/9/15 p=1
|
||||
B: O1 ✓ (3587 vs 3899) O2 ✗ (95/33/78 p=0.22) O3 ✓ (viol 0, solved 205 vs 205) O4 ✓ -> does NOT transfer
|
||||
vs brent total 3587 vs 6576 per problem 125/36/45 p=6.6e-10
|
||||
vs toms748 total 3587 vs 5914 per problem 150/28/28 p=2.3e-21
|
||||
per class (ours / Chandrupatla, per problem w/t/l):
|
||||
C0: 178 / 156 2/1/3 p=1
|
||||
C1: 513 / 474 17/7/16 p=1
|
||||
C2: 459 / 437 15/10/15 p=1
|
||||
C3: 1294 / 1792 36/1/3 p=3.6e-08
|
||||
C4: 515 / 466 9/5/26 p=0.006
|
||||
C5: 628 / 574 16/9/15 p=1
|
||||
|
||||
ABLATION delta = (without) - (with); + = the component earns its place
|
||||
A - G2 APS -8 CHA -3 NEW +82 earned
|
||||
A - contra APS +9 CHA +22 NEW -8 earned
|
||||
A - half APS +214 CHA +26 NEW +97 earned
|
||||
A - exp APS +0 CHA +63 NEW +145 earned
|
||||
A - point APS +2 CHA +442 NEW +770 earned
|
||||
A - regime APS +6 CHA +999 NEW +1830 earned
|
||||
B - defect APS +284 CHA +67 NEW +17 earned
|
||||
B - exp APS +1 CHA +192 NEW +126 earned
|
||||
B - point APS -1 CHA +558 NEW +740 earned
|
||||
B - regime APS -8 CHA +1238 NEW +1911 earned
|
||||
oos done
|
||||
File diff suppressed because one or more lines are too long
+165
@@ -0,0 +1,165 @@
|
||||
"""PREREG_out_of_sample_2026-09-21: the two strongest variants, their leave-one-out ablations
|
||||
and the three library rivals, on APS, CHA and the out-of-sample corpus (corpus_new.py).
|
||||
|
||||
python -u sweep_oos.py run -> sweep_oos.json, then the verdicts
|
||||
python -u sweep_oos.py --report verdicts only, from sweep_oos.json
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from multiprocessing import Pool
|
||||
|
||||
from corpus_new import build as build_new
|
||||
from sweep_full import ACC, MAXFEV, XTOL, load_problems, make_method
|
||||
|
||||
P = "project"
|
||||
|
||||
|
||||
def spec(pre, ext="none"):
|
||||
return ("pure", "ratio", "derived3", P, "x", pre, "recent", "distance", "none", ext)
|
||||
|
||||
|
||||
CONFIGS = {
|
||||
"base": spec("none"),
|
||||
"A": spec("contra+flatanalogmempt+exp+half", "pattern"),
|
||||
"A-G2": spec("contra+flatanalogmempt+exp+half"),
|
||||
"A-contra": spec("flatanalogmempt+exp+half", "pattern"),
|
||||
"A-half": spec("contra+flatanalogmempt+exp", "pattern"),
|
||||
"A-exp": spec("contra+flatanalogmempt+half", "pattern"),
|
||||
"A-point": spec("contra+flatanalogmem+exp+half", "pattern"),
|
||||
"A-regime": spec("contra+half", "pattern"),
|
||||
"B": spec("defect+flatanalogmempt+exp"),
|
||||
"B-defect": spec("flatanalogmempt+exp"),
|
||||
"B-exp": spec("defect+flatanalogmempt"),
|
||||
"B-point": spec("defect+flatanalogmem+exp"),
|
||||
"B-regime": spec("defect"),
|
||||
"chandrupatla": ("base", "chandrupatla"),
|
||||
"brent": ("base", "brent"),
|
||||
"toms748": ("base", "toms748"),
|
||||
}
|
||||
|
||||
|
||||
def problems():
|
||||
out = [(s, pid, f, a, b, r) for s, pid, f, a, b, r in load_problems()]
|
||||
new, _ = build_new()
|
||||
out += [("NEW", pid, f, a, b, r) for _cls, pid, f, a, b, r in new]
|
||||
return out
|
||||
|
||||
|
||||
def _reset_random_problem():
|
||||
"""SciPy's original.05.00 (our C0.05) is a random step function: unseeded random() values
|
||||
kept in a module-level cache shared by every run in the process. Addendum to the PREREG
|
||||
(after two unseeded runs gave the same verdicts): clear the cache and seed before EVERY
|
||||
run, so each method meets the same function and the run is reproducible."""
|
||||
import random
|
||||
|
||||
import scipy.optimize._tstutils as T
|
||||
T._f6_cache.clear()
|
||||
random.seed(20260921)
|
||||
|
||||
|
||||
def evaluate(name):
|
||||
method = make_method(CONFIGS[name])
|
||||
res = {}
|
||||
for s, pid, f, a, b, root in problems():
|
||||
_reset_random_problem()
|
||||
try:
|
||||
x, n, conv = method(f, a, b, XTOL, MAXFEV)
|
||||
except Exception:
|
||||
x, n, conv = float("nan"), MAXFEV, False
|
||||
ok = bool(conv and math.isfinite(x) and abs(x - root) <= ACC)
|
||||
over = ok and int(n) > 2 * math.ceil(math.log2(abs(b - a) / XTOL)) + 4
|
||||
res.setdefault(s, {})[pid] = [int(n), ok, bool(over)]
|
||||
return name, res
|
||||
|
||||
|
||||
def total(r, s, keep=None):
|
||||
return sum((n if ok else MAXFEV) for pid, (n, ok, _o) in r[s].items()
|
||||
if keep is None or keep(pid))
|
||||
|
||||
|
||||
def h2h(r, o, s, keep=None):
|
||||
w = t = l = 0
|
||||
for pid, (n, ok, _o) in r[s].items():
|
||||
if keep is not None and not keep(pid):
|
||||
continue
|
||||
c1 = n if ok else MAXFEV
|
||||
n2, ok2, _ = o[s][pid]
|
||||
c2 = n2 if ok2 else MAXFEV
|
||||
w += c1 < c2
|
||||
t += c1 == c2
|
||||
l += c1 > c2
|
||||
return w, t, l
|
||||
|
||||
|
||||
def sign_p(w, l):
|
||||
n = w + l
|
||||
if n == 0:
|
||||
return 1.0
|
||||
k = min(w, l)
|
||||
return min(1.0, 2.0 * sum(math.comb(n, i) for i in range(k + 1)) / 2.0 ** n)
|
||||
|
||||
|
||||
def report(R):
|
||||
ch = R["chandrupatla"]
|
||||
print("TOTALS APS CHA NEW (NEW C0-C2 NEW C3-C5)")
|
||||
neutral = lambda pid: pid[:2] in ("C0", "C1", "C2")
|
||||
ours_home = lambda pid: pid[:2] in ("C3", "C4", "C5")
|
||||
for k in ("base", "A", "B", "chandrupatla", "brent", "toms748"):
|
||||
r = R[k]
|
||||
print(f" {k:<18} {total(r, 'APS'):6d} {total(r, 'CHA'):6d} {total(r, 'NEW'):6d}"
|
||||
f" ({total(r, 'NEW', neutral):6d} {total(r, 'NEW', ours_home):6d})")
|
||||
print("\nOUT OF SAMPLE (NEW) - verdicts")
|
||||
for v in ("A", "B"):
|
||||
r = R[v]
|
||||
o1 = total(r, "NEW") < total(ch, "NEW")
|
||||
w, t, l = h2h(r, ch, "NEW")
|
||||
o2 = w > l and sign_p(w, l) < 0.05
|
||||
viol = sum(o for _n, _ok, o in r["NEW"].values())
|
||||
solved = sum(ok for _n, ok, _o in r["NEW"].values())
|
||||
solved_ch = sum(ok for _n, ok, _o in ch["NEW"].values())
|
||||
o3 = viol == 0 and solved >= solved_ch
|
||||
o4 = total(r, "NEW") < total(R["base"], "NEW")
|
||||
print(f" {v}: O1 {'✓' if o1 else '✗'} ({total(r, 'NEW')} vs {total(ch, 'NEW')}) "
|
||||
f"O2 {'✓' if o2 else '✗'} ({w}/{t}/{l} p={sign_p(w, l):.2g}) "
|
||||
f"O3 {'✓' if o3 else '✗'} (viol {viol}, solved {solved} vs {solved_ch}) "
|
||||
f"O4 {'✓' if o4 else '✗'} -> "
|
||||
+ ("TRANSFERS" if o1 and o2 and o3 else "does NOT transfer"))
|
||||
for rn in ("brent", "toms748"):
|
||||
w2, t2, l2 = h2h(r, R[rn], "NEW")
|
||||
print(f" vs {rn:<8} total {total(r, 'NEW')} vs {total(R[rn], 'NEW')} "
|
||||
f"per problem {w2}/{t2}/{l2} p={sign_p(w2, l2):.2g}")
|
||||
print(" per class (ours / Chandrupatla, per problem w/t/l):")
|
||||
for cls in ("C0", "C1", "C2", "C3", "C4", "C5"):
|
||||
keep = lambda pid, c=cls: pid.startswith(c)
|
||||
w3, t3, l3 = h2h(r, ch, "NEW", keep)
|
||||
print(f" {cls}: {total(r, 'NEW', keep):5d} / {total(ch, 'NEW', keep):5d}"
|
||||
f" {w3}/{t3}/{l3} p={sign_p(w3, l3):.2g}")
|
||||
print("\nABLATION delta = (without) - (with); + = the component earns its place")
|
||||
for v, parts in (("A", ("G2", "contra", "half", "exp", "point", "regime")),
|
||||
("B", ("defect", "exp", "point", "regime"))):
|
||||
r = R[v]
|
||||
for p in parts:
|
||||
o = R[f"{v}-{p}"]
|
||||
d = [total(o, s) - total(r, s) for s in ("APS", "CHA", "NEW")]
|
||||
earned = any(x > 0 for x in d)
|
||||
print(f" {v} - {p:<7} APS {d[0]:+5d} CHA {d[1]:+5d} NEW {d[2]:+5d} "
|
||||
+ ("earned" if earned else "NOT earned"))
|
||||
|
||||
|
||||
def main():
|
||||
if "--report" not in sys.argv:
|
||||
with Pool(min(8, len(CONFIGS))) as pool:
|
||||
R = dict(pool.map(evaluate, list(CONFIGS)))
|
||||
with open("sweep_oos.json", "w", encoding="utf8") as fh:
|
||||
json.dump(R, fh)
|
||||
with open("sweep_oos.json", encoding="utf8") as fh:
|
||||
R = json.load(fh)
|
||||
report(R)
|
||||
print("oos done")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user