TOTALS (evaluations of f, failures charged 200) method APS CHA both proposed rule 2806 3220 6026 + G2 (step pattern) 2782 3169 5951 + G2 + analogy entry 2780 2625 5405 + contra + remembered analogy 2792 2537 5329 + defect + remembered analogy 2530 2561 5091 + G2 + analogy + ORDER POINT 2780 2221 5001 + G2 + contra + rem. + ORDER PT 2775 2119 4894 + defect + rem. + ORDER POINT 2532 2164 4696 + G2 + contra + rem. + pt + EXP 2773 2073 4846 + defect + rem. + pt + EXP 2531 1972 4503 + half (alone) 2564 3202 5766 + G2 + contra + rem + pt + exp + HALF 2559 2047 4606 + contra + rem + pt + exp + HALF 2551 2044 4595 Chandrupatla 2787 2441 5228 Brent 2889 4115 7004 TOMS748 2820 4000 6820 PER PROBLEM, ours against each rival: wins/ties/losses (sign-test p) APS proposed rule Chandrupatla 87/24/42 p=9.2e-05 Brent 102/11/40 p=2e-07 TOMS748 86/16/51 p=0.0035 + G2 (step pattern) Chandrupatla 95/20/38 p=8.2e-07 Brent 110/9/34 p=1.5e-10 TOMS748 97/15/41 p=2.1e-06 + G2 + analogy entry Chandrupatla 95/20/38 p=8.2e-07 Brent 110/9/34 p=1.5e-10 TOMS748 97/15/41 p=2.1e-06 + contra + remembered analogy Chandrupatla 90/23/40 p=1.4e-05 Brent 105/11/37 p=9.9e-09 TOMS748 87/16/50 p=0.002 + defect + remembered analogy Chandrupatla 86/20/47 p=0.00091 Brent 101/13/39 p=1.6e-07 TOMS748 107/11/35 p=1.1e-09 + G2 + analogy + ORDER POINT Chandrupatla 95/20/38 p=8.2e-07 Brent 110/9/34 p=1.5e-10 TOMS748 97/15/41 p=2.1e-06 + G2 + contra + rem. + ORDER PT Chandrupatla 98/18/37 p=1.5e-07 Brent 112/8/33 p=3e-11 TOMS748 98/15/40 p=8.5e-07 + defect + rem. + ORDER POINT Chandrupatla 86/20/47 p=0.00091 Brent 101/12/40 p=2.9e-07 TOMS748 107/11/35 p=1.1e-09 + G2 + contra + rem. + pt + EXP Chandrupatla 98/18/37 p=1.5e-07 Brent 112/8/33 p=3e-11 TOMS748 98/15/40 p=8.5e-07 + defect + rem. + pt + EXP Chandrupatla 86/20/47 p=0.00091 Brent 101/12/40 p=2.9e-07 TOMS748 107/11/35 p=1.1e-09 + half (alone) Chandrupatla 98/18/37 p=1.5e-07 Brent 116/7/30 p=3.9e-13 TOMS748 117/9/27 p=1.5e-14 + G2 + contra + rem + pt + exp + HALF Chandrupatla 98/17/38 p=2.7e-07 Brent 114/8/31 p=2.3e-12 TOMS748 118/7/28 p=2.4e-14 + contra + rem + pt + exp + HALF Chandrupatla 99/16/38 p=1.9e-07 Brent 115/8/30 p=6.2e-13 TOMS748 117/8/28 p=3.9e-14 CHA proposed rule Chandrupatla 9/5/26 p=0.006 Brent 30/4/4 p=6.2e-06 TOMS748 31/1/8 p=0.00029 + G2 (step pattern) Chandrupatla 10/3/27 p=0.0076 Brent 30/2/6 p=7e-05 TOMS748 31/1/8 p=0.00029 + G2 + analogy entry Chandrupatla 9/6/25 p=0.009 Brent 30/2/6 p=7e-05 TOMS748 36/1/3 p=3.6e-08 + contra + remembered analogy Chandrupatla 8/7/25 p=0.0046 Brent 31/4/3 p=7.7e-07 TOMS748 37/1/2 p=2.8e-09 + defect + remembered analogy Chandrupatla 8/5/27 p=0.0019 Brent 30/2/6 p=7e-05 TOMS748 35/1/4 p=3.4e-07 + G2 + analogy + ORDER POINT Chandrupatla 24/3/13 p=0.099 Brent 30/2/6 p=7e-05 TOMS748 35/1/4 p=3.4e-07 + G2 + contra + rem. + ORDER PT Chandrupatla 22/3/15 p=0.32 Brent 31/2/5 p=1.3e-05 TOMS748 36/1/3 p=3.6e-08 + defect + rem. + ORDER POINT Chandrupatla 20/3/17 p=0.74 Brent 30/2/6 p=7e-05 TOMS748 34/1/5 p=2.4e-06 + G2 + contra + rem. + pt + EXP Chandrupatla 28/3/9 p=0.0026 Brent 31/3/4 p=3.5e-06 TOMS748 37/1/2 p=2.8e-09 + defect + rem. + pt + EXP Chandrupatla 27/3/10 p=0.0076 Brent 30/2/6 p=7e-05 TOMS748 35/1/4 p=3.4e-07 + half (alone) Chandrupatla 8/6/26 p=0.0029 Brent 33/3/2 p=3.7e-08 TOMS748 33/0/7 p=4.2e-05 + G2 + contra + rem + pt + exp + HALF Chandrupatla 27/6/7 p=0.00082 Brent 33/3/2 p=3.7e-08 TOMS748 38/1/1 p=1.5e-10 + contra + rem + pt + exp + HALF Chandrupatla 27/6/7 p=0.00082 Brent 34/4/0 p=1.2e-10 TOMS748 38/1/1 p=1.5e-10 CHA PER FAMILY (ours / Chandrupatla) fun1 fun2 fun3 fun4 fun5 fun6 fun7 fun8 fun9 proposed rule 99 141 338 573 456 456 1000 52 105 + G2 (step pattern) 97 146 346 522 456 456 1000 52 94 + G2 + analogy entry 99 146 336 331 254 313 1000 52 94 + contra + remembered analogy 108 141 304 318 254 275 1000 52 85 + defect + remembered analogy 123 144 304 315 257 280 1000 57 81 + G2 + analogy + ORDER POINT 82 146 230 261 100 256 1000 52 94 + G2 + contra + rem. + ORDER PT 94 146 144 143 252 208 1000 52 80 + defect + rem. + ORDER POINT 99 144 132 141 177 339 1000 57 75 + G2 + contra + rem. + pt + EXP 98 146 147 158 208 187 1000 52 77 + defect + rem. + pt + EXP 100 144 123 146 161 167 1000 57 74 + half (alone) 106 141 353 548 452 456 1000 52 94 + G2 + contra + rem + pt + exp + HALF 98 147 147 173 208 154 1000 52 68 + contra + rem + pt + exp + HALF 103 141 145 173 208 154 1000 52 68 Chandrupatla 103 152 305 297 233 233 1000 63 55