OK number of configurations got 264 want 264 The proposed method (projection, multiplier declared): OK APS place got 3 want 3 OK APS total got 2806 want 2806 OK APS solved got 153 want 153 OK CHA place got 45 want 45 OK CHA total got 3220 want 3220 OK CHA solved got 40 want 40 OK both sets got 6026 want 6026 Head to head of the proposed method: OK APS vs chandrupatla got (87, 24, 42) want (87, 24, 42) OK APS vs toms748 got (86, 16, 51) want (86, 16, 51) OK APS vs brent got (102, 11, 40) want (102, 11, 40) OK APS vs itp got (132, 3, 18) want (132, 3, 18) OK APS vs itp_oracle got (121, 4, 28) want (121, 4, 28) OK APS vs bisection got (152, 1, 0) want (152, 1, 0) OK CHA vs chandrupatla got (9, 5, 26) want (9, 5, 26) OK CHA vs toms748 got (31, 1, 8) want (31, 1, 8) OK CHA vs brent got (30, 4, 4) want (30, 4, 4) OK CHA vs itp got (17, 0, 23) want (17, 0, 23) OK CHA vs itp_oracle got (16, 0, 24) want (16, 0, 24) OK APS vs Chandrupatla sign test p < 1e-4 got True want True OK CHA vs Chandrupatla sign test p (rounded) got 0.006 want 0.006 The constant-free variant (earned projection): OK credit APS place got 29 want 29 OK credit APS total got 3568 want 3568 OK credit CHA place got 10 want 10 OK credit CHA total got 3020 want 3020 OK credit both sets got 6588 want 6588 OK credit costs 762 more on the canonical set got 762 want 762 OK credit APS vs chandrupatla got (79, 3, 71) want (79, 3, 71) OK credit APS vs toms748 got (77, 9, 67) want (77, 9, 67) OK credit APS vs brent got (78, 5, 70) want (78, 5, 70) OK credit APS vs Chandrupatla sign test p (rounded) got 0.57 want 0.57 The guards that replace the step instead of projecting it: OK APS unguarded got 2806 want 2806 OK CHA unguarded total got 4104 want 4104 OK CHA unguarded solved got 35 want 35 OK CHA unguarded bound violations got 10 want 10 OK APS bracket-guard got 7260 want 7260 OK APS both-guard got 7239 want 7239 Alefeld-Potra-Shi: OK APS winner got 'chandrupatla' want 'chandrupatla' OK APS chandrupatla place got 1 want 1 OK APS chandrupatla total got 2787 want 2787 OK APS toms748 place got 4 want 4 OK APS toms748 total got 2820 want 2820 OK APS brent place got 5 want 5 OK APS brent total got 2889 want 2889 OK APS dekker place got 12 want 12 OK APS dekker total got 3158 want 3158 OK APS itp_oracle place got 13 want 13 OK APS itp_oracle total got 3260 want 3260 OK APS ridders place got 20 want 20 OK APS ridders total got 3313 want 3313 OK APS itp place got 34 want 34 OK APS itp total got 3957 want 3957 OK APS illinois place got 73 want 73 OK APS illinois total got 7045 want 7045 OK APS bisection place got 79 want 79 OK APS bisection total got 7530 want 7530 OK APS regula_falsi place got 244 want 244 OK APS regula_falsi total got 10560 want 10560 OK APS regula falsi solved got 127 want 127 Chandrupatla: OK CHA rank 1 label got 'pure.sqrt.bracket.project' want 'pure.sqrt.bracket.project' OK CHA rank 1 solves one more than anyone else got 41 want 41 OK CHA chandrupatla place got 2 want 2 OK CHA chandrupatla total got 2441 want 2441 OK CHA itp_oracle place got 3 want 3 OK CHA itp_oracle total got 2647 want 2647 OK CHA itp place got 6 want 6 OK CHA itp total got 2896 want 2896 OK CHA bisection place got 16 want 16 OK CHA bisection total got 3090 want 3090 OK CHA toms748 place got 159 want 159 OK CHA toms748 total got 4000 want 4000 OK CHA brent place got 227 want 227 OK CHA brent total got 4115 want 4115 OK CHA brent solved got 38 want 38 OK chandrupatla, both sets got 5228 want 5228 OK Chandrupatla beats us on both sets got True want True Negative results, the regime claims: OK CHA cbrt over the whole set got 4109 want 4109 OK single fixed choice, both sets got 6026 want 6026 OK switch by benchmark set, both sets got 6915 want 6915 OK switching is worse than not switching got True want True OK fun4.1 projection variant got 89 want 89 OK fun4.1 cbrt got 32 want 32 OK fun4.2 projection variant got 93 want 93 OK fun4.2 cbrt got 32 want 32 OK fun4.3 projection variant got 113 want 113 OK fun4.3 cbrt got 40 want 40 OK fun4.4 projection variant got 125 want 125 OK fun4.4 cbrt got 47 want 47 OK fun4.5 projection variant got 153 want 153 OK fun4.5 cbrt got 59 want 59 The law across the grid: OK mean total, scale=derived3, lin got 6893 want 6893 OK mean total, scale=derived3, quad got 22315 want 22315 OK mean total, scale=derived, lin got 6981 want 6981 OK mean total, scale=derived, quad got 22206 want 22206 OK mean total, scale=bracket, lin got 10279 want 10279 OK mean total, scale=bracket, quad got 19127 want 19127 OK mean total, scale=start, lin got 16102 want 16102 OK mean total, scale=start, quad got 15345 want 15345 OK every c=1 similarity beats every c=0 similarity got True want True OK similarity ranking got ['ratio', 'exp_abs', 'rational', 'ratio_soft', 'sqrt', 'cbrt', 'exp_sq', 'rat_sq', 'cosine'] want ['ratio', 'exp_abs', 'rational', 'ratio_soft', 'sqrt', 'cbrt', 'exp_sq', 'rat_sq', 'cosine'] OK similarity ratio best got 6560 want 6560 OK similarity exp_abs best got 6895 want 6895 OK similarity rational best got 7118 want 7118 OK similarity ratio_soft best got 9578 want 9578 OK similarity sqrt best got 10546 want 10546 OK similarity cbrt best got 12002 want 12002 OK similarity exp_sq best got 14307 want 14307 OK similarity rat_sq best got 15102 want 15102 OK similarity cosine best got 16394 want 16394 OK scale ranking got ['derived3', 'derived', 'bracket', 'start'] want ['derived3', 'derived', 'bracket', 'start'] OK scale derived3 best got 6560 want 6560 OK scale derived best got 6702 want 6702 OK scale bracket best got 9107 want 9107 OK scale start best got 14540 want 14540 The placement modes (each in its own sweep file): OK slope.ratio.derived3.project APS got 2848 want 2848 OK slope.ratio.derived3.project CHA got 3114 want 3114 OK slope.ratio.derived3.project solved got 193 want 193 OK slope.ratio.derived3.project guarantee (over 2x) got 0 want 0 OK pure+shape+slope.ratio.derived3.safe APS got 2527 want 2527 OK pure+shape+slope.ratio.derived3.safe CHA got 3126 want 3126 OK pure+shape+slope.ratio.derived3.safe solved got 193 want 193 OK pure+shape+slope.ratio.derived3.safe guarantee (over 2x) got 0 want 0 OK pure+dissim+slope.ratio.derived3.project APS got 2726 want 2726 OK pure+dissim+slope.ratio.derived3.project CHA got 2799 want 2799 OK pure+dissim+slope.ratio.derived3.project solved got 193 want 193 OK pure+dissim+slope.ratio.derived3.project guarantee (over 2x) got 0 want 0 OK pure+shape+dissim+slope.ratio.derived3.safe APS got 2884 want 2884 OK pure+shape+dissim+slope.ratio.derived3.safe CHA got 2871 want 2871 OK pure+shape+dissim+slope.ratio.derived3.safe solved got 193 want 193 OK pure+shape+dissim+slope.ratio.derived3.safe guarantee (over 2x) got 0 want 0 OK progress.ratio.derived.project APS got 4449 want 4449 OK progress.ratio.derived.project CHA got 3047 want 3047 OK progress.ratio.derived.project solved got 193 want 193 OK progress.ratio.derived.project guarantee (over 2x) got 0 want 0 OK log.ratio_soft.derived.safe APS got 6940 want 6940 OK log.ratio_soft.derived.safe CHA got 3541 want 3541 OK log.ratio_soft.derived.safe solved got 193 want 193 OK log.ratio_soft.derived.safe guarantee (over 2x) got 0 want 0 OK pure+dissim+slope.ratio.derived3.safe CHA per-set got 2695 want 2695 OK progress.ratio_soft.derived.safe CHA per-set got 2888 want 2888 The per-place reading (sweep_order.json): OK far+rational both APS m=1 got 4522 want 4522 OK far+rational both APS order got 3675 want 3675 OK far+rational both CHA m=1 got 3406 want 3406 OK far+rational both CHA order got 3241 want 3241 OK far+rational bracket APS m=1 got 4606 want 4606 OK far+rational bracket APS order got 3542 want 3542 OK far+rational bracket CHA m=1 got 3602 want 3602 OK far+rational bracket CHA order got 3359 want 3359 OK far+rational credit APS m=1 got 5190 want 5190 OK far+rational credit APS order got 4686 want 4686 OK far+rational credit CHA m=1 got 3013 want 3013 OK far+rational credit CHA order got 2884 want 2884 OK far+rational project APS m=1 got 6932 want 6932 OK far+rational project APS order got 6364 want 6364 OK far+rational project CHA m=1 got 4789 want 4789 OK far+rational project CHA order got 4554 want 4554 OK far+rational safe APS m=1 got 5987 want 5987 OK far+rational safe APS order got 5229 want 5229 OK far+rational safe CHA m=1 got 4134 want 4134 OK far+rational safe CHA order got 3954 want 3954 OK far+rational step APS m=1 got 6435 want 6435 OK far+rational step APS order got 5551 want 5551 OK far+rational step CHA m=1 got 4287 want 4287 OK far+rational step CHA order got 4068 want 4068 OK far+rational none APS m=1 got 10676 want 10676 OK far+rational none APS order got 10043 want 10043 OK far+rational none CHA m=1 got 8265 want 8265 OK far+rational none CHA order got 7794 want 7794 OK cheaper in 7 of 7 safeguards got 7 want 7 OK false position none solved got 127 want 127 OK false position none APS got 10676 want 10676 OK false position order solved got 128 want 128 OK false position order APS got 10043 want 10043 OK slope reading none APS got 2806 want 2806 OK slope reading order APS got 2854 want 2854 OK guarantee family violations got 0 want 0 The decision place (sweep_decision.json): OK ratio.band.project APS got 5432 want 5432 OK ratio.band.project CHA got 3059 want 3059 OK ratio.band.credit CHA got 2892 want 2892 OK ratio.none.project APS got 2806 want 2806 OK ratio.defect.project APS got 2523 want 2523 OK ratio.defect.project CHA got 3210 want 3210 OK ratio_soft.simpair.project CHA got 2897 want 2897 OK defect APS solved got 153 want 153 OK defect both sets got 5733 want 5733 OK defect is the new canonical best (< 2527) got True want True OK decision grid guarantee violations got 0 want 0 OK band refusal rate % got 63.5 want 63.5 OK defect refusal rate % got 6.6 want 6.6 The gap decomposition (fresh runs of the diagnosis scripts): OK problems where we lose got 33 want 33 OK fun1.4 our bracket halvings/eval got 0.62 want 0.62 OK fun1.4 IQI from our points got 10.37 want 10.37 OK flat: ours min got 0.41 want 0.41 OK flat: ours max got 0.75 want 0.75 OK flat: IQI min got 0.01 want 0.01 OK flat: IQI max got 0.17 want 0.17 OK non-degenerate states with one negative weight got np.int64(428) want 428 OK bracketing states in total got 431 want 431 OK every exception is an underflowed zero weight got True want True OK total gap on the losing problems got 256 want 256 OK flat families' gap got 172 want 172 OK fun1 gap got 58 want 58 OK Chandrupatla bisect share fun3 % got 92 want 92 OK Chandrupatla bisect share fun4 % got 96 want 96 OK Chandrupatla bisect share fun5 % got 100 want 100 OK Chandrupatla bisect share fun6 % got 100 want 100 OK Chandrupatla total fun4 got 297 want 297 OK Chandrupatla total fun5 got 233 want 233 OK Chandrupatla total fun6 got 233 want 233 OK m>=2 on canonical states % got np.float64(0.2) want 0.2 OK m>=2 on flat states % got 0.0 want 0.0 OK band fun4 no-gate got 573 want 573 OK band fun4 with band got 296 want 296 OK band fun5 no-gate got 456 want 456 OK band fun5 with band got 239 want 239 OK band fun6 no-gate got 456 want 456 OK band fun6 with band got 228 want 228 OK band cost on fun1 got 164 want 164 OK band cost on fun2 got 111 want 111 OK band cost on fun8 got 170 want 170 OK band cost on fun9 got 144 want 144 OK log item: x-space control APS got 5822 want 5822 OK log item: x-space control CHA got 3695 want 3695 §3.1: the multiplier and the endowment (fresh runs): OK multiplier 1.0 APS got 4817 want 4817 OK multiplier 1.0 CHA got 2997 want 2997 OK multiplier 1.25 APS got 2806 want 2806 OK multiplier 1.25 CHA got 2752 want 2752 OK multiplier 1.5 APS got 2806 want 2806 OK multiplier 1.5 CHA got 2889 want 2889 OK multiplier 2.0 APS got 2806 want 2806 OK multiplier 2.0 CHA got 3220 want 3220 OK multiplier 3.0 APS got 2806 want 2806 OK multiplier 3.0 CHA got 3756 want 3756 OK multiplier 4.0 APS got 2806 want 2806 OK multiplier 4.0 CHA got 4006 want 4006 OK endowment 0.1 APS got 5344 want 5344 OK endowment 0.1 CHA got 3051 want 3051 OK endowment 0.5 APS got 4656 want 4656 OK endowment 0.5 CHA got 3002 want 3002 OK endowment 1.0 APS got 3568 want 3568 OK endowment 1.0 CHA got 3020 want 3020 OK endowment 2.0 APS got 3692 want 3692 OK endowment 2.0 CHA got 2712 want 2712 OK endowment 5.0 APS got 3006 want 3006 OK endowment 5.0 CHA got 2615 want 2615 OK C0=1 cheapest within the one-step promise got 1.0 want 1.0 OK 2527 at multiplier 1.25 got 2541 want 2541 OK 2527 flat from 1.5 to 4 got {2527} want {2527} OK 2523 flat from 1.25 to 4 got {2523} want {2523} Tables 2-3 regenerated: OK APS pure.ratio.derived3.project place got 3 want 3 OK APS pure.ratio.derived3.none place got 2 want 2 OK CHA pure.ratio.derived3.project place got 45 want 45 OK CHA pure.ratio.derived3.none place got 228 want 228 OK CHA pure.exp_abs.bracket.both place got 4 want 4 OK CHA exp_abs.bracket.both total got 2736 want 2736 OK CHA TOMS 748 over the 2x bound got 15 want 15 OK CHA Brent over the 2x bound got 18 want 18 OK credit APS vs itp_oracle got (86, 6, 61) want (86, 6, 61) OK credit CHA vs itp_oracle got (2, 2, 36) want (2, 2, 36) OK credit CHA vs bisection got (5, 0, 35) want (5, 0, 35) OK project APS vs chandrupatla got (87, 24, 42) want (87, 24, 42) OK project APS vs itp got (132, 3, 18) want (132, 3, 18) OK project APS vs itp_oracle got (121, 4, 28) want (121, 4, 28) OK project APS vs bisection got (152, 1, 0) want (152, 1, 0) OK project CHA vs toms748 got (31, 1, 8) want (31, 1, 8) OK project CHA vs brent got (30, 4, 4) want (30, 4, 4) OK project CHA vs itp got (17, 0, 23) want (17, 0, 23) OK project CHA vs itp_oracle got (16, 0, 24) want (16, 0, 24) OK both sets pure.ratio.derived3.credit got 6588 want 6588 OK both sets pure.ratio.derived3.project got 6026 want 6026 OK both sets chandrupatla got 5228 want 5228 OK both sets itp_oracle got 5907 want 5907 OK both sets toms748 got 6820 want 6820 OK both sets brent got 7004 want 7004 §7 under the earned projection: OK slope.ratio.derived3.credit APS got 3507 want 3507 OK slope.ratio.derived3.credit CHA got 2997 want 2997 OK pure+shape+slope.ratio.derived3.credit APS got 3755 want 3755 OK pure+shape+slope.ratio.derived3.credit CHA got 3021 want 3021 OK pure+dissim+slope.ratio.derived3.credit APS got 3857 want 3857 OK pure+dissim+slope.ratio.derived3.credit CHA got 3002 want 3002 OK pure+shape+dissim+slope.ratio.derived3.credit APS got 4354 want 4354 OK pure+shape+dissim+slope.ratio.derived3.credit CHA got 3037 want 3037 OK defect.ratio.derived3.credit APS got 3545 want 3545 OK defect.ratio.derived3.credit CHA got 2949 want 2949 OK best constant-free configuration got 'defect+flatanalogmempt+exp+half.ratio.derived3.credit' want 'defect+flatanalogmempt+exp+half.ratio.derived3.credit' OK its total over both sets got 6111 want 6111 OK defect under the earned projection, both sets got 6494 want 6494 Section 8 and negative items 12-16 (the development grids): OK flat.ratio.derived3.project APS got 3029 want 3029 OK flat.ratio.derived3.project CHA got 3198 want 3198 OK flat.ratio.derived3.project guarantee got 0 want 0 OK flat2.ratio.derived3.project APS got 2803 want 2803 OK flat2.ratio.derived3.project CHA got 3341 want 3341 OK flat2.ratio.derived3.project guarantee got 0 want 0 OK flatcreep.ratio.derived3.project APS got 2878 want 2878 OK flatcreep.ratio.derived3.project CHA got 3354 want 3354 OK flatcreep.ratio.derived3.project guarantee got 0 want 0 OK flatstickmid.ratio.derived3.project APS got 6897 want 6897 OK flatstickmid.ratio.derived3.project CHA got 3019 want 3019 OK flatstickmid.ratio.derived3.project guarantee got 0 want 0 OK flatstickexit.ratio.derived3.project APS got 3125 want 3125 OK flatstickexit.ratio.derived3.project CHA got 2503 want 2503 OK flatstickexit.ratio.derived3.project guarantee got 0 want 0 OK flatjudge.ratio.derived3.project APS got 3009 want 3009 OK flatjudge.ratio.derived3.project CHA got 2486 want 2486 OK flatjudge.ratio.derived3.project guarantee got 0 want 0 OK flatanalogmem.ratio.derived3.project APS got 2814 want 2814 OK flatanalogmem.ratio.derived3.project guarantee got 0 want 0 OK half.ratio.derived3.project APS got 2564 want 2564 OK half.ratio.derived3.project guarantee got 0 want 0 OK sticky with order from the failed promise, APS got 6781 want 6781 OK sticky midpoint: flat families got 1057 want 1057 OK bisection: flat families got 1044 want 1044 OK Chandrupatla: flat families got 1068 want 1068 OK split from the order: flat families, low end got 743 want 743 OK split from the order: flat families, high end got 847 want 847 OK item 16: remembered + split, with the step pattern, CHA got 2117 want 2117 OK item 16: one-state analogy + split, with the step pattern, CHA got 2221 want 2221 Section 8 diagnostics (fresh runs): OK flag rises on canonical problems got 142 want 142 OK flag + analogy + m>=2 on canonical problems got 3 want 3 OK flag + analogy + m>=2 on flat problems got 20 want 20 OK flag + analogy + m>=2 on fun2 got 0 want 0 OK flag + analogy + m>=2 on fun8 got 0 want 0 OK flag + analogy + m>=2 on fun9 got 0 want 0 OK mean entry state, one state got '12.9' want '12.9' OK mean entry state, remembered got '8.2' want '8.2' OK order from the failed promise is inf got (163, 174) want (163, 174) OK sqrt|x| sgn x, base rule got 84 want 84 OK sin on [3,4], base rule got 24 want 24 OK sin on [3,4], A' got 7 want 7 Table 8 (A' on the protocol of section 4, sweep_aprime.json): OK A' APS got 2438 want 2438 OK A' CHA got 2005 want 2005 OK A' both got 4443 want 4443 OK A' solved got 193 want 193 OK A' earned budget got (3608, 2807) want (3608, 2807) OK A' under every safeguard: violations got 0 want 0 OK A' safeguards measured got 7 want 7 OK A' APS vs chandrupatla got (103, 20, 30) want (103, 20, 30) OK A' CHA vs chandrupatla got (29, 4, 7) want (29, 4, 7) OK A' APS vs toms748 got (127, 6, 20) want (127, 6, 20) OK A' CHA vs toms748 got (38, 2, 0) want (38, 2, 0) OK A' APS vs brent got (118, 17, 18) want (118, 17, 18) OK A' CHA vs brent got (33, 3, 2) want (33, 3, 2) OK A' APS vs itp_oracle got (126, 7, 20) want (126, 7, 20) OK A' CHA vs itp_oracle got (35, 1, 4) want (35, 1, 4) OK A' APS vs Chandrupatla p (2 digits) got 1.5e-10 want 1.5e-10 OK A' CHA vs Chandrupatla p (1 digit) got 0.0003 want 0.0003 OK A' below Chandrupatla, APS % got 12.5 want 12.5 OK A' below Chandrupatla, CHA % got 17.9 want 17.9 OK A' fun9 vs Chandrupatla got (65, 55) want (65, 55) OK A' ahead on wide-bracket families got 7 want 7 OK A' level on fun7 got True want True Table 9 (ablation of A' on six corpora, sweep_ablation_all.json): OK ablation -regime got (-7, 1013, 1660, 324, 77, 2269) want (-7, 1013, 1660, 324, 77, 2269) OK ablation -point got (-5, 479, 667, 272, 30, 1672) want (-5, 479, 667, 272, 30, 1672) OK ablation -sub got (-6, 0, -7, 16, 63, 893) want (-6, 0, -7, 16, 63, 893) OK ablation -close got (130, 35, 97, 78, 135, -2) want (130, 35, 97, 78, 135, -2) OK ablation -half got (249, 65, 82, -16, -5, 1) want (249, 65, 82, -16, -5, 1) OK ablation -exp got (31, 143, 100, -21, -2, 87) want (31, 143, 100, -21, -2, 87) OK ablation -G2 got (-22, -6, 123, -30, -10, 34) want (-22, -6, 123, -30, -10, 34) OK ablation -contra got (11, -12, 29, 0, 1, -1) want (11, -12, 29, 0, 1, -1) Section 9 (out-of-sample corpora): OK corpus APS size got 154 want 154 OK corpus CHA size got 45 want 45 OK corpus NEW size got 206 want 206 OK corpus RJL size got 78 want 78 OK corpus APP size got 132 want 132 OK corpus D size got 240 want 240 OK Table 10 APS totals got (2259, 2595, 2707, 2636) want (2259, 2595, 2707, 2636) OK Table 10 APS vs Chandrupatla got (103, 20, 31) want (103, 20, 31) OK Table 10 APS A' violations got 0 want 0 OK Table 10 CHA totals got (1074, 1505, 3238, 3088) want (1074, 1505, 3238, 3088) OK Table 10 CHA vs Chandrupatla got (31, 4, 10) want (31, 4, 10) OK Table 10 CHA A' violations got 0 want 0 OK Table 10 NEW totals got (3290, 3716, 6395, 5746) want (3290, 3716, 6395, 5746) OK Table 10 NEW vs Chandrupatla got (108, 33, 65) want (108, 33, 65) OK Table 10 NEW A' violations got 0 want 0 OK Table 10 RJL totals got (1371, 1566, 2455, 2339) want (1371, 1566, 2455, 2339) OK Table 10 RJL vs Chandrupatla got (44, 14, 20) want (44, 14, 20) OK Table 10 RJL A' violations got 0 want 0 OK Table 10 APP totals got (1382, 1382, 1503, 1756) want (1382, 1382, 1503, 1756) OK Table 10 APP vs Chandrupatla got (48, 52, 32) want (48, 52, 32) OK Table 10 APP A' violations got 0 want 0 OK Table 10 D totals got (3656, 5717, 8267, 8084) want (3656, 5717, 8267, 8084) OK Table 10 D vs Chandrupatla got (177, 35, 28) want (177, 35, 28) OK Table 10 D A' violations got 0 want 0 OK Brent / TOMS 748 bound violations over 855 got (118, 84) want (118, 84) OK all 855 totals got (13032, 16481, 24565, 23649) want (13032, 16481, 24565, 23649) OK corpus 4 without targeted got (2403, 3569, (107, 31, 22)) want (2403, 3569, (107, 31, 22)) OK corpus 4 simple roots got (737, 747, (43, 22, 15)) want (737, 747, (43, 22, 15)) OK Table 11 D1 got (433, 419, (20, 8, 12)) want (433, 419, (20, 8, 12)) OK Table 11 D2 got (1163, 1805, (39, 1, 0)) want (1163, 1805, (39, 1, 0)) OK Table 11 D3 got (924, 1784, (37, 1, 2)) want (924, 1784, (37, 1, 2)) OK Table 11 D4 got (304, 328, (23, 14, 3)) want (304, 328, (23, 14, 3)) OK Table 11 D5 got (503, 1017, (25, 8, 7)) want (503, 1017, (25, 8, 7)) OK Table 11 D6 got (329, 364, (33, 3, 4)) want (329, 364, (33, 3, 4)) OK near-cancelling class of corpus 1 got (491, 391) want (491, 391) OK corpus 1, variant carried: total vs Chandrupatla got (3492, 3899) want (3492, 3899) OK corpus 1, variant carried: per problem got (105, 34, 67) want (105, 34, 67) OK corpus 1, with defect: per problem got (95, 33, 78) want (95, 33, 78) OK corpus 1, with defect: exponential class got (9, 5, 26) want (9, 5, 26) OK Roots.jl, variant carried: totals got (1464, 1566) want (1464, 1566) OK Roots.jl, variant carried: per problem got (38, 10, 30) want (38, 10, 30) OK applied, variant carried: totals got (1580, 1382) want (1580, 1382) ModAB (sweep_modab.json, the authors' package): OK ModAB protocol APS, CHA, both got (2914, 2436, 5350) want (2914, 2436, 5350) OK A' vs ModAB protocol APS got (118, 11, 25) want (118, 11, 25) OK A' vs ModAB protocol CHA got (32, 9, 4) want (32, 9, 4) OK Table 10 ModAB APS got (2726, 0) want (2726, 0) OK Table 10 A' vs ModAB APS got (118, 10, 26) want (118, 10, 26) OK Table 10 ModAB CHA got (1681, 0) want (1681, 0) OK Table 10 A' vs ModAB CHA got (34, 4, 7) want (34, 4, 7) OK Table 10 ModAB NEW got (3809, 1) want (3809, 1) OK Table 10 A' vs ModAB NEW got (125, 26, 55) want (125, 26, 55) OK Table 10 ModAB RJL got (1446, 0) want (1446, 0) OK Table 10 A' vs ModAB RJL got (47, 11, 20) want (47, 11, 20) OK Table 10 ModAB APP got (1460, 0) want (1460, 0) OK Table 10 A' vs ModAB APP got (76, 35, 21) want (76, 35, 21) OK Table 10 ModAB D got (5535, 8) want (5535, 8) OK Table 10 A' vs ModAB D got (183, 23, 34) want (183, 23, 34) OK A' vs ModAB: every corpus p <= 0.0013 got True want True OK ModAB over all 855 got 16657 want 16657 OK ModAB violations over 855 got 9 want 9 OK corpus 4 A' vs ModAB p (1 digit) got '7e-26' want '7e-26' OK ModAB's own benchmark, all 91 got (91, 1507, 1588, (56, 13, 22)) want (91, 1507, 1588, (56, 13, 22)) OK ModAB's own benchmark p (2 digits) got '0.00015' want '0.00015' Table 9a and section 7: the similarity in the step (fresh audit run): OK Table 9a ratio APS+CHA got 4443 want 4443 OK Table 9a ratio six corpora got 13032 want 13032 OK Table 9a ratio guarantee got 0 want 0 OK Table 9a exp_abs APS+CHA got 4988 want 4988 OK Table 9a exp_abs six corpora got 14142 want 14142 OK Table 9a exp_abs guarantee got 0 want 0 OK Table 9a rational APS+CHA got 5486 want 5486 OK Table 9a rational six corpora got 14843 want 14843 OK Table 9a rational guarantee got 0 want 0 OK Table 9a ratio_soft APS+CHA got 8360 want 8360 OK Table 9a ratio_soft six corpora got 28500 want 28500 OK Table 9a ratio_soft guarantee got 0 want 0 OK Table 9a sqrt APS+CHA got 12718 want 12718 OK Table 9a sqrt six corpora got 45552 want 45552 OK Table 9a sqrt guarantee got 0 want 0 OK Table 9a cbrt APS+CHA got 17207 want 17207 OK Table 9a cbrt six corpora got 63031 want 63031 OK Table 9a cbrt guarantee got 0 want 0 OK Table 9a exp_sq APS+CHA got 18410 want 18410 OK Table 9a exp_sq six corpora got 67701 want 67701 OK Table 9a exp_sq guarantee got 0 want 0 OK Table 9a cosine APS+CHA got 18412 want 18412 OK Table 9a cosine six corpora got 67854 want 67854 OK Table 9a cosine guarantee got 0 want 0 OK Table 9a rat_sq APS+CHA got 18422 want 18422 OK Table 9a rat_sq six corpora got 67394 want 67394 OK Table 9a rat_sq guarantee got 0 want 0 OK nearest other S costs more, protocol % got 12 want 12 OK worst S costs more, protocol % got 315 want 315 OK nearest other S costs more, six corpora % got 9 want 9 OK quadratic S: four to five times got (4, 5) want (4, 5) OK audit: shape gain with the tent (APS) got -493 want -493 OK audit: shape gain with 1/sqrt(1+z^2) (APS) got 250 want 250 OK audit: defect gain with the tent (APS) got -283 want -283 OK audit: defect gain with 1-|z|^1/3 (APS) got 215 want 215 OK sign test p for 103:31 got '3e-10' want '3e-10' OK sign test p for 31:10 got '0.001' want '0.001' OK sign test p for 108:65 got '0.001' want '0.001' OK sign test p for 44:20 got '0.004' want '0.004' OK sign test p for 48:32 got '0.09' want '0.09' OK sign test p for 177:28 got '1e-27' want '1e-27' OK sign test p for 107:22 got '1e-14' want '1e-14' OK sign test p for 43:15 got '0.0003' want '0.0003' OK sign test p for 105:67 got '0.005' want '0.005' OK sign test p for 38:30 got '0.4' want '0.4' every number stated in the paper matches the sweep data.