374 lines
30 KiB
Plaintext
374 lines
30 KiB
Plaintext
OK number of configurations got 264 want 264
|
|
|
|
The proposed method (projection, multiplier declared):
|
|
OK APS place got 3 want 3
|
|
OK APS total got 2806 want 2806
|
|
OK APS solved got 153 want 153
|
|
BAD CHA place got 45 want 47
|
|
OK CHA total got 3220 want 3220
|
|
OK CHA solved got 40 want 40
|
|
OK both sets got 6026 want 6026
|
|
|
|
Head to head of the proposed method:
|
|
OK APS vs chandrupatla got (87, 24, 42) want (87, 24, 42)
|
|
OK APS vs toms748 got (86, 16, 51) want (86, 16, 51)
|
|
OK APS vs brent got (102, 11, 40) want (102, 11, 40)
|
|
OK APS vs itp got (132, 3, 18) want (132, 3, 18)
|
|
OK APS vs itp_oracle got (121, 4, 28) want (121, 4, 28)
|
|
OK APS vs bisection got (152, 1, 0) want (152, 1, 0)
|
|
OK CHA vs chandrupatla got (9, 5, 26) want (9, 5, 26)
|
|
OK CHA vs toms748 got (31, 1, 8) want (31, 1, 8)
|
|
OK CHA vs brent got (30, 4, 4) want (30, 4, 4)
|
|
OK CHA vs itp got (17, 0, 23) want (17, 0, 23)
|
|
OK CHA vs itp_oracle got (16, 0, 24) want (16, 0, 24)
|
|
OK APS vs Chandrupatla sign test p < 1e-4 got True want True
|
|
OK CHA vs Chandrupatla sign test p (rounded) got 0.006 want 0.006
|
|
|
|
The constant-free variant (earned projection):
|
|
OK credit APS place got 29 want 29
|
|
OK credit APS total got 3568 want 3568
|
|
OK credit CHA place got 10 want 10
|
|
OK credit CHA total got 3020 want 3020
|
|
OK credit both sets got 6588 want 6588
|
|
OK credit costs 762 more on the canonical set got 762 want 762
|
|
OK credit APS vs chandrupatla got (79, 3, 71) want (79, 3, 71)
|
|
OK credit APS vs toms748 got (77, 9, 67) want (77, 9, 67)
|
|
OK credit APS vs brent got (78, 5, 70) want (78, 5, 70)
|
|
OK credit APS vs Chandrupatla sign test p (rounded) got 0.57 want 0.57
|
|
|
|
The guards that replace the step instead of projecting it:
|
|
OK APS unguarded got 2806 want 2806
|
|
OK CHA unguarded total got 4104 want 4104
|
|
OK CHA unguarded solved got 35 want 35
|
|
OK CHA unguarded bound violations got 10 want 10
|
|
OK APS bracket-guard got 7260 want 7260
|
|
OK APS both-guard got 7239 want 7239
|
|
|
|
Alefeld-Potra-Shi:
|
|
OK APS winner got 'chandrupatla' want 'chandrupatla'
|
|
OK APS chandrupatla place got 1 want 1
|
|
OK APS chandrupatla total got 2787 want 2787
|
|
OK APS toms748 place got 4 want 4
|
|
OK APS toms748 total got 2820 want 2820
|
|
OK APS brent place got 5 want 5
|
|
OK APS brent total got 2889 want 2889
|
|
OK APS dekker place got 12 want 12
|
|
OK APS dekker total got 3158 want 3158
|
|
OK APS itp_oracle place got 13 want 13
|
|
OK APS itp_oracle total got 3260 want 3260
|
|
OK APS ridders place got 20 want 20
|
|
OK APS ridders total got 3313 want 3313
|
|
OK APS itp place got 34 want 34
|
|
OK APS itp total got 3957 want 3957
|
|
OK APS illinois place got 73 want 73
|
|
OK APS illinois total got 7045 want 7045
|
|
OK APS bisection place got 79 want 79
|
|
OK APS bisection total got 7530 want 7530
|
|
OK APS regula_falsi place got 244 want 244
|
|
OK APS regula_falsi total got 10560 want 10560
|
|
OK APS regula falsi solved got 127 want 127
|
|
|
|
Chandrupatla:
|
|
OK CHA rank 1 label got 'pure.sqrt.bracket.project' want 'pure.sqrt.bracket.project'
|
|
OK CHA rank 1 solves one more than anyone else got 41 want 41
|
|
OK CHA chandrupatla place got 2 want 2
|
|
OK CHA chandrupatla total got 2441 want 2441
|
|
OK CHA itp_oracle place got 3 want 3
|
|
BAD CHA itp_oracle total got 2647 want 2611
|
|
OK CHA itp place got 6 want 6
|
|
BAD CHA itp total got 2896 want 2857
|
|
OK CHA bisection place got 16 want 16
|
|
OK CHA bisection total got 3090 want 3090
|
|
BAD CHA toms748 place got 159 want 164
|
|
OK CHA toms748 total got 4000 want 4000
|
|
BAD CHA brent place got 227 want 226
|
|
OK CHA brent total got 4115 want 4115
|
|
OK CHA brent solved got 38 want 38
|
|
OK chandrupatla, both sets got 5228 want 5228
|
|
OK Chandrupatla beats us on both sets got True want True
|
|
|
|
Negative results, the regime claims:
|
|
BAD CHA cbrt over the whole set got 4109 want 4099
|
|
OK single fixed choice, both sets got 6026 want 6026
|
|
BAD switch by benchmark set, both sets got 6915 want 6905
|
|
OK switching is worse than not switching got True want True
|
|
OK fun4.1 projection variant got 89 want 89
|
|
OK fun4.1 cbrt got 32 want 32
|
|
OK fun4.2 projection variant got 93 want 93
|
|
OK fun4.2 cbrt got 32 want 32
|
|
OK fun4.3 projection variant got 113 want 113
|
|
OK fun4.3 cbrt got 40 want 40
|
|
OK fun4.4 projection variant got 125 want 125
|
|
OK fun4.4 cbrt got 47 want 47
|
|
OK fun4.5 projection variant got 153 want 153
|
|
OK fun4.5 cbrt got 59 want 59
|
|
|
|
The law across the grid:
|
|
BAD mean total, scale=derived3, lin got 6893 want 6809
|
|
BAD mean total, scale=derived3, quad got 22315 want 22294
|
|
BAD mean total, scale=derived, lin got 6981 want 6906
|
|
BAD mean total, scale=derived, quad got 22206 want 22167
|
|
BAD mean total, scale=bracket, lin got 10279 want 10263
|
|
BAD mean total, scale=bracket, quad got 19127 want 19137
|
|
OK mean total, scale=start, lin got 16102 want 16102
|
|
OK mean total, scale=start, quad got 15345 want 15345
|
|
OK every c=1 similarity beats every c=0 similarity got True want True
|
|
OK similarity ranking got ['ratio', 'exp_abs', 'rational', 'ratio_soft', 'sqrt', 'cbrt', 'exp_sq', 'rat_sq', 'cosine'] want ['ratio', 'exp_abs', 'rational', 'ratio_soft', 'sqrt', 'cbrt', 'exp_sq', 'rat_sq', 'cosine']
|
|
BAD similarity ratio best got 6560 want 6496
|
|
BAD similarity exp_abs best got 6895 want 6822
|
|
BAD similarity rational best got 7118 want 7038
|
|
BAD similarity ratio_soft best got 9578 want 9502
|
|
BAD similarity sqrt best got 10546 want 10497
|
|
BAD similarity cbrt best got 12002 want 11941
|
|
BAD similarity exp_sq best got 14307 want 14314
|
|
OK similarity rat_sq best got 15102 want 15102
|
|
OK similarity cosine best got 16394 want 16394
|
|
OK scale ranking got ['derived3', 'derived', 'bracket', 'start'] want ['derived3', 'derived', 'bracket', 'start']
|
|
BAD scale derived3 best got 6560 want 6496
|
|
BAD scale derived best got 6702 want 6628
|
|
BAD scale bracket best got 9107 want 9081
|
|
OK scale start best got 14540 want 14540
|
|
|
|
The placement modes (each in its own sweep file):
|
|
OK slope.ratio.derived3.project APS got 2848 want 2848
|
|
BAD slope.ratio.derived3.project CHA got 3114 want 3074
|
|
OK slope.ratio.derived3.project solved got 193 want 193
|
|
OK slope.ratio.derived3.project guarantee (over 2x) got 0 want 0
|
|
OK pure+shape+slope.ratio.derived3.safe APS got 2527 want 2527
|
|
BAD pure+shape+slope.ratio.derived3.safe CHA got 3126 want 3108
|
|
OK pure+shape+slope.ratio.derived3.safe solved got 193 want 193
|
|
OK pure+shape+slope.ratio.derived3.safe guarantee (over 2x) got 0 want 0
|
|
OK pure+dissim+slope.ratio.derived3.project APS got 2726 want 2726
|
|
BAD pure+dissim+slope.ratio.derived3.project CHA got 2799 want 2785
|
|
OK pure+dissim+slope.ratio.derived3.project solved got 193 want 193
|
|
OK pure+dissim+slope.ratio.derived3.project guarantee (over 2x) got 0 want 0
|
|
OK pure+shape+dissim+slope.ratio.derived3.safe APS got 2884 want 2884
|
|
BAD pure+shape+dissim+slope.ratio.derived3.safe CHA got 2871 want 2859
|
|
OK pure+shape+dissim+slope.ratio.derived3.safe solved got 193 want 193
|
|
OK pure+shape+dissim+slope.ratio.derived3.safe guarantee (over 2x) got 0 want 0
|
|
!! gate.ratio.derived.project not in sweep_gate.json — stale sweep
|
|
OK log.ratio_soft.derived.safe APS got 6940 want 6940
|
|
BAD log.ratio_soft.derived.safe CHA got 3541 want 3474
|
|
OK log.ratio_soft.derived.safe solved got 193 want 193
|
|
OK log.ratio_soft.derived.safe guarantee (over 2x) got 0 want 0
|
|
BAD pure+dissim+slope.ratio.derived3.safe CHA per-set got 2695 want 2683
|
|
|
|
The per-place reading (sweep_order.json):
|
|
OK far+rational both APS m=1 got 4522 want 4522
|
|
OK far+rational both APS order got 3675 want 3675
|
|
BAD far+rational both CHA m=1 got 3406 want 3387
|
|
BAD far+rational both CHA order got 3241 want 3216
|
|
OK far+rational bracket APS m=1 got 4606 want 4606
|
|
OK far+rational bracket APS order got 3542 want 3542
|
|
BAD far+rational bracket CHA m=1 got 3602 want 3571
|
|
BAD far+rational bracket CHA order got 3359 want 3303
|
|
OK far+rational credit APS m=1 got 5190 want 5190
|
|
OK far+rational credit APS order got 4686 want 4686
|
|
OK far+rational credit CHA m=1 got 3013 want 3013
|
|
OK far+rational credit CHA order got 2884 want 2884
|
|
OK far+rational project APS m=1 got 6932 want 6932
|
|
OK far+rational project APS order got 6364 want 6364
|
|
BAD far+rational project CHA m=1 got 4789 want 4794
|
|
BAD far+rational project CHA order got 4554 want 4559
|
|
OK far+rational safe APS m=1 got 5987 want 5987
|
|
OK far+rational safe APS order got 5229 want 5229
|
|
BAD far+rational safe CHA m=1 got 4134 want 4108
|
|
BAD far+rational safe CHA order got 3954 want 3923
|
|
OK far+rational step APS m=1 got 6435 want 6435
|
|
OK far+rational step APS order got 5551 want 5551
|
|
BAD far+rational step CHA m=1 got 4287 want 4257
|
|
BAD far+rational step CHA order got 4068 want 4022
|
|
OK far+rational none APS m=1 got 10676 want 10676
|
|
OK far+rational none APS order got 10043 want 10043
|
|
OK far+rational none CHA m=1 got 8265 want 8265
|
|
OK far+rational none CHA order got 7794 want 7794
|
|
OK cheaper in 7 of 7 safeguards got 7 want 7
|
|
OK false position none solved got 127 want 127
|
|
OK false position none APS got 10676 want 10676
|
|
OK false position order solved got 128 want 128
|
|
OK false position order APS got 10043 want 10043
|
|
OK slope reading none APS got 2806 want 2806
|
|
OK slope reading order APS got 2854 want 2854
|
|
OK guarantee family violations got 0 want 0
|
|
|
|
The decision place (sweep_decision.json):
|
|
OK ratio.band.project APS got 5432 want 5432
|
|
BAD ratio.band.project CHA got 3059 want 3066
|
|
BAD ratio.band.credit CHA got 2892 want 2899
|
|
OK ratio.none.project APS got 2806 want 2806
|
|
OK ratio.defect.project APS got 2523 want 2523
|
|
OK ratio.defect.project CHA got 3210 want 3210
|
|
BAD ratio_soft.simpair.project CHA got 2897 want 2868
|
|
OK defect APS solved got 153 want 153
|
|
OK defect both sets got 5733 want 5733
|
|
OK defect is the new canonical best (< 2527) got True want True
|
|
OK decision grid guarantee violations got 0 want 0
|
|
BAD band refusal rate % got 63.5 want 63.6
|
|
BAD defect refusal rate % got 6.6 want 7.0
|
|
|
|
The gap decomposition (fresh runs of the diagnosis scripts):
|
|
OK problems where we lose got 33 want 33
|
|
OK fun1.4 our bracket halvings/eval got 0.62 want 0.62
|
|
OK fun1.4 IQI from our points got 10.37 want 10.37
|
|
OK flat: ours min got 0.41 want 0.41
|
|
BAD flat: ours max got 0.75 want 0.81
|
|
OK flat: IQI min got 0.01 want 0.01
|
|
OK flat: IQI max got 0.17 want 0.17
|
|
OK non-degenerate states with one negative weight got np.int64(428) want 428
|
|
BAD bracketing states in total got 431 want 433
|
|
OK every exception is an underflowed zero weight got True want True
|
|
BAD total gap on the losing problems got 256 want 244
|
|
BAD flat families' gap got 172 want 160
|
|
OK fun1 gap got 58 want 58
|
|
OK Chandrupatla bisect share fun3 % got 92 want 92
|
|
OK Chandrupatla bisect share fun4 % got 96 want 96
|
|
OK Chandrupatla bisect share fun5 % got 100 want 100
|
|
OK Chandrupatla bisect share fun6 % got 100 want 100
|
|
OK Chandrupatla total fun4 got 297 want 297
|
|
OK Chandrupatla total fun5 got 233 want 233
|
|
OK Chandrupatla total fun6 got 233 want 233
|
|
OK m>=2 on canonical states % got np.float64(0.2) want 0.2
|
|
BAD m>=2 on flat states % got 0.0 want 5.8
|
|
OK band fun4 no-gate got 573 want 573
|
|
OK band fun4 with band got 296 want 296
|
|
OK band fun5 no-gate got 456 want 456
|
|
OK band fun5 with band got 239 want 239
|
|
OK band fun6 no-gate got 456 want 456
|
|
BAD band fun6 with band got 228 want 235
|
|
OK band cost on fun1 got 164 want 164
|
|
OK band cost on fun2 got 111 want 111
|
|
OK band cost on fun8 got 170 want 170
|
|
OK band cost on fun9 got 144 want 144
|
|
OK log item: x-space control APS got 5822 want 5822
|
|
BAD log item: x-space control CHA got 3695 want 3643
|
|
|
|
§3.1: the multiplier and the endowment (fresh runs):
|
|
OK multiplier 1.0 APS got 4817 want 4817
|
|
OK multiplier 1.0 CHA got 2997 want 2997
|
|
OK multiplier 1.25 APS got 2806 want 2806
|
|
OK multiplier 1.25 CHA got 2752 want 2752
|
|
OK multiplier 1.5 APS got 2806 want 2806
|
|
OK multiplier 1.5 CHA got 2889 want 2889
|
|
OK multiplier 2.0 APS got 2806 want 2806
|
|
OK multiplier 2.0 CHA got 3220 want 3220
|
|
OK multiplier 3.0 APS got 2806 want 2806
|
|
OK multiplier 3.0 CHA got 3756 want 3756
|
|
OK multiplier 4.0 APS got 2806 want 2806
|
|
OK multiplier 4.0 CHA got 4006 want 4006
|
|
OK endowment 0.1 APS got 5344 want 5344
|
|
OK endowment 0.1 CHA got 3051 want 3051
|
|
OK endowment 0.5 APS got 4656 want 4656
|
|
OK endowment 0.5 CHA got 3002 want 3002
|
|
OK endowment 1.0 APS got 3568 want 3568
|
|
OK endowment 1.0 CHA got 3020 want 3020
|
|
OK endowment 2.0 APS got 3692 want 3692
|
|
OK endowment 2.0 CHA got 2712 want 2712
|
|
OK endowment 5.0 APS got 3006 want 3006
|
|
OK endowment 5.0 CHA got 2615 want 2615
|
|
OK C0=1 cheapest within the one-step promise got 1.0 want 1.0
|
|
OK 2527 at multiplier 1.25 got 2541 want 2541
|
|
OK 2527 flat from 1.5 to 4 got {2527} want {2527}
|
|
OK 2523 flat from 1.25 to 4 got {2523} want {2523}
|
|
|
|
Tables 2-3 regenerated:
|
|
OK APS pure.ratio.derived3.project place got 3 want 3
|
|
OK APS pure.ratio.derived3.none place got 2 want 2
|
|
BAD CHA pure.ratio.derived3.project place got 45 want 47
|
|
OK CHA pure.ratio.derived3.none place got 228 want 228
|
|
OK CHA pure.exp_abs.bracket.both place got 4 want 4
|
|
BAD CHA exp_abs.bracket.both total got 2736 want 2733
|
|
OK CHA TOMS 748 over the 2x bound got 15 want 15
|
|
OK CHA Brent over the 2x bound got 18 want 18
|
|
OK credit APS vs itp_oracle got (86, 6, 61) want (86, 6, 61)
|
|
BAD credit CHA vs itp_oracle got (2, 2, 36) want (2, 3, 35)
|
|
OK credit CHA vs bisection got (5, 0, 35) want (5, 0, 35)
|
|
OK project APS vs chandrupatla got (87, 24, 42) want (87, 24, 42)
|
|
OK project APS vs itp got (132, 3, 18) want (132, 3, 18)
|
|
OK project APS vs itp_oracle got (121, 4, 28) want (121, 4, 28)
|
|
OK project APS vs bisection got (152, 1, 0) want (152, 1, 0)
|
|
OK project CHA vs toms748 got (31, 1, 8) want (31, 1, 8)
|
|
OK project CHA vs brent got (30, 4, 4) want (30, 4, 4)
|
|
OK project CHA vs itp got (17, 0, 23) want (17, 0, 23)
|
|
OK project CHA vs itp_oracle got (16, 0, 24) want (16, 0, 24)
|
|
OK both sets pure.ratio.derived3.credit got 6588 want 6588
|
|
OK both sets pure.ratio.derived3.project got 6026 want 6026
|
|
OK both sets chandrupatla got 5228 want 5228
|
|
BAD both sets itp_oracle got 5907 want 5871
|
|
OK both sets toms748 got 6820 want 6820
|
|
OK both sets brent got 7004 want 7004
|
|
|
|
§7 under the earned projection:
|
|
OK slope.ratio.derived3.credit APS got 3507 want 3507
|
|
OK slope.ratio.derived3.credit CHA got 2997 want 2997
|
|
OK pure+shape+slope.ratio.derived3.credit APS got 3755 want 3755
|
|
OK pure+shape+slope.ratio.derived3.credit CHA got 3021 want 3021
|
|
OK pure+dissim+slope.ratio.derived3.credit APS got 3857 want 3857
|
|
OK pure+dissim+slope.ratio.derived3.credit CHA got 3002 want 3002
|
|
OK pure+shape+dissim+slope.ratio.derived3.credit APS got 4354 want 4354
|
|
OK pure+shape+dissim+slope.ratio.derived3.credit CHA got 3037 want 3037
|
|
OK defect.ratio.derived3.credit APS got 3545 want 3545
|
|
OK defect.ratio.derived3.credit CHA got 2949 want 2949
|
|
BAD best constant-free configuration got 'defect+flatanalogmempt+exp+half.ratio.derived3.credit' want 'defect.ratio.derived3.credit'
|
|
BAD its total over both sets got 6111 want 6494
|
|
|
|
59 STATED NUMBERS DO NOT MATCH THE DATA:
|
|
CHA place
|
|
CHA itp_oracle total
|
|
CHA itp total
|
|
CHA toms748 place
|
|
CHA brent place
|
|
CHA cbrt over the whole set
|
|
switch by benchmark set, both sets
|
|
mean total, scale=derived3, lin
|
|
mean total, scale=derived3, quad
|
|
mean total, scale=derived, lin
|
|
mean total, scale=derived, quad
|
|
mean total, scale=bracket, lin
|
|
mean total, scale=bracket, quad
|
|
similarity ratio best
|
|
similarity exp_abs best
|
|
similarity rational best
|
|
similarity ratio_soft best
|
|
similarity sqrt best
|
|
similarity cbrt best
|
|
similarity exp_sq best
|
|
scale derived3 best
|
|
scale derived best
|
|
scale bracket best
|
|
slope.ratio.derived3.project CHA
|
|
pure+shape+slope.ratio.derived3.safe CHA
|
|
pure+dissim+slope.ratio.derived3.project CHA
|
|
pure+shape+dissim+slope.ratio.derived3.safe CHA
|
|
gate.ratio.derived.project missing from sweep_gate.json
|
|
log.ratio_soft.derived.safe CHA
|
|
pure+dissim+slope.ratio.derived3.safe CHA per-set
|
|
gate.ratio_soft.derived.safe missing from sweep_gate.json
|
|
far+rational both CHA m=1
|
|
far+rational both CHA order
|
|
far+rational bracket CHA m=1
|
|
far+rational bracket CHA order
|
|
far+rational project CHA m=1
|
|
far+rational project CHA order
|
|
far+rational safe CHA m=1
|
|
far+rational safe CHA order
|
|
far+rational step CHA m=1
|
|
far+rational step CHA order
|
|
ratio.band.project CHA
|
|
ratio.band.credit CHA
|
|
ratio_soft.simpair.project CHA
|
|
band refusal rate %
|
|
defect refusal rate %
|
|
flat: ours max
|
|
bracketing states in total
|
|
total gap on the losing problems
|
|
flat families' gap
|
|
m>=2 on flat states %
|
|
band fun6 with band
|
|
log item: x-space control CHA
|
|
CHA pure.ratio.derived3.project place
|
|
CHA exp_abs.bracket.both total
|
|
credit CHA vs itp_oracle
|
|
both sets itp_oracle
|
|
best constant-free configuration
|
|
its total over both sets
|