PREREG_out_of_sample outcome (server). Variant A (G2 + contra + remembered analogy + order point + exp limit + half) TRANSFERS: new corpus 3492 vs Chandrupatla 3899, per problem 105/34/67 p=0.0046, guarantee held, solved 205 = 205. Variant B (defect + remembered + point + exp) does NOT (total 3587 below, but per problem 95/33/78 p=0.22; loses the exponential class 9/5/26). Per class the transfer is carried by the multiple/flat-root class C3 (1194 vs 1792, 37/0/3); on C0-C2 A is at par per problem and ~6% dearer in total, C4 ~ par, C5 (near-cancelling) 691 vs 574. Ablation: every component earns its place on at least one set (contra only on APS/CHA: +9/+22, NEW -8; G2 only on NEW: +82). Addendum: C0.05 is SciPy's random step function - seeded and its cache cleared before every run for reproducibility; two unseeded runs gave the same verdicts.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-09-21 19:33:07 +01:00
parent 47361dabea
commit a86a9ed263
4 changed files with 217 additions and 0 deletions
@@ -0,0 +1,42 @@
+TOTALS                 APS    CHA    NEW  (NEW C0-C2  NEW C3-C5)
+  base                 2806   3220   5572  (  1208    4364)
+  A                    2559   2047   3492  (  1133    2359)
+  B                    2531   1972   3587  (  1150    2437)
+  chandrupatla         2787   2441   3899  (  1067    2832)
+  brent                2889   4115   6576  (  1196    5380)
+  toms748              2820   4000   5914  (  1251    4663)
+
+OUT OF SAMPLE (NEW) - verdicts
+  A: O1 ✓ (3492 vs 3899)  O2 ✓ (105/34/67 p=0.0046)  O3 ✓ (viol 0, solved 205 vs 205)  O4 ✓  -> TRANSFERS
+       vs brent    total 3492 vs 6576  per problem 139/34/33 p=1.1e-16
+       vs toms748  total 3492 vs 5914  per problem 159/27/20 p=4.6e-28
+       per class (ours / Chandrupatla, per problem w/t/l):
+         C0:   173 /   156   2/1/3 p=1
+         C1:   524 /   474   18/5/17 p=1
+         C2:   436 /   437   16/14/10 p=0.33
+         C3:  1194 /  1792   37/0/3 p=1.9e-08
+         C4:   474 /   466   16/5/19 p=0.74
+         C5:   691 /   574   16/9/15 p=1
+  B: O1 ✓ (3587 vs 3899)  O2 ✗ (95/33/78 p=0.22)  O3 ✓ (viol 0, solved 205 vs 205)  O4 ✓  -> does NOT transfer
+       vs brent    total 3587 vs 6576  per problem 125/36/45 p=6.6e-10
+       vs toms748  total 3587 vs 5914  per problem 150/28/28 p=2.3e-21
+       per class (ours / Chandrupatla, per problem w/t/l):
+         C0:   178 /   156   2/1/3 p=1
+         C1:   513 /   474   17/7/16 p=1
+         C2:   459 /   437   15/10/15 p=1
+         C3:  1294 /  1792   36/1/3 p=3.6e-08
+         C4:   515 /   466   9/5/26 p=0.006
+         C5:   628 /   574   16/9/15 p=1
+
+ABLATION  delta = (without) - (with); + = the component earns its place
+  A - G2      APS    -8  CHA    -3  NEW   +82   earned
+  A - contra  APS    +9  CHA   +22  NEW    -8   earned
+  A - half    APS  +214  CHA   +26  NEW   +97   earned
+  A - exp     APS    +0  CHA   +63  NEW  +145   earned
+  A - point   APS    +2  CHA  +442  NEW  +770   earned
+  A - regime  APS    +6  CHA  +999  NEW +1830   earned
+  B - defect  APS  +284  CHA   +67  NEW   +17   earned
+  B - exp     APS    +1  CHA  +192  NEW  +126   earned
+  B - point   APS    -1  CHA  +558  NEW  +740   earned
+  B - regime  APS    -8  CHA +1238  NEW +1911   earned
+oos done