diff --git a/article/sim_numerics.pdf b/article/sim_numerics.pdf index be66730..41423f0 100644 Binary files a/article/sim_numerics.pdf and b/article/sim_numerics.pdf differ diff --git a/article/sim_numerics.tex b/article/sim_numerics.tex index cee9801..289e334 100644 --- a/article/sim_numerics.tex +++ b/article/sim_numerics.tex @@ -563,37 +563,61 @@ The readings do not compose. Taken together, (2) and (3) give $2884$ and $2859$ --- worse than either alone. Two selections that are each an improvement interfere, which is a reason to measure composition rather than assume it. -\subsection{Five problems of the wide-bracket set are scored unsolved for every -method} +\subsection{Six problems of the two sets are scored unsolved for every method} \label{sec:fun7} -Chasing the remaining gap to Chandrupatla on the wide-bracket set led to a -property of the set itself. On the five problems of its \texttt{fun7} family the -declared root is $0$, but $f$ vanishes \emph{identically} on a neighbourhood of -it: $f(0+t)=0$ for every $t$ up to at least $3.2\times10^{-2}$. Every method -therefore stops after $8$ to $22$ evaluations at a point where $f(x)=0$ -\emph{exactly} --- a genuine root --- and every method is then scored as having -failed, because the harness asks for $|x-\text{root}|\le10^{-6}$ and the -returned point sits up to $2.8\times10^{-2}$ away. +Chasing the remaining gap to Chandrupatla led to a property of the benchmarks +themselves. On the five problems of the wide-bracket family \texttt{fun7}, and on +the single canonical problem \texttt{aps.13.00}, the declared root is $0$ while +$f$ vanishes \emph{identically} on a neighbourhood of it in floating point: +$f(0\pm t)=0$ for every $t$ up to at least $3.2\times10^{-2}$, because the +exponential factor underflows --- at the returned points $|f(x)|$ is below the +smallest normal double. Every method therefore stops after $8$ to $28$ +evaluations at a point where $f(x)=0$ \emph{exactly} --- a genuine root --- and +every method is then scored as having failed, because the harness asks for +$|x-\text{root}|\le10^{-6}$ while the returned point sits up to +$2.8\times10^{-2}$ away. The consequence is arithmetic. Each such problem charges the full budget of -$200$ evaluations to every method, so $1000$ evaluations of every wide-bracket -total --- ours, Chandrupatla's, ITP's, bisection's --- are this artefact, and the -``$40$ of $45$ solved'' that every method reports is a statement about the -scoring rather than about the methods. Re-scored with the criterion ``a root is a -root'' ($|x-\text{root}|\le10^{-6}$ \emph{or} $f(x)=0$), the same runs give -$45/45$ for every method except Brent's $43/45$, and the totals fall to -Chandrupatla $1487$, ours $1740$, ITP $1907$, bisection $2136$, TOMS\,748 -$3088$. The ranking does not change; the magnitudes do, and with them the size -of the remaining gap: $253$ evaluations, or $17\%$, not $11\%$. +$200$ evaluations to every method: about $1150$ evaluations of every reported +two-set total --- ours, Chandrupatla's, TOMS\,748's, ITP's, Brent's, bisection's +--- are this artefact, and the ``$153$ of $154$'' and ``$40$ of $45$'' that every +method reports are statements about the scoring rather than about the methods. +Re-scored with the criterion ``a root is a root'' +($|x-\text{root}|\le10^{-6}$ \emph{or} $f(x)=0$), the same runs solve $199$ of +$199$ for every method except Brent's $197$, and the totals fall as follows. + +\begin{table}[h] +\centering +\begin{tabular}{lrrrr} +\hline + & \multicolumn{2}{c}{APS-154} & \multicolumn{2}{c}{CHA-45} \\ +method & as scored & re-scored & as scored & re-scored \\ +\hline +ours, points by shape & $\mathbf{2527}$ & $\mathbf{2337}$ & $3108$ & $2159$ \\ +ours, projection variant & $2806$ & $2634$ & $3220$ & $2357$ \\ +Chandrupatla & $2785$ & $2593$ & $\mathbf{2423}$ & $\mathbf{1487}$ \\ +TOMS\,748 & $2820$ & $2636$ & $4000$ & $3088$ \\ +Brent & $2889$ & $2707$ & $4099$ & $3222$ \\ +ITP & $3957$ & $3767$ & $2857$ & $1907$ \\ +bisection & $7530$ & $7338$ & $3090$ & $2136$ \\ +\hline +\end{tabular} +\caption{The same runs under the harness criterion and under ``a root is a +root''. The ranking is unchanged; the magnitudes are not, and neither is the size +of the remaining gap on the wide-bracket set: $253$ evaluations, or $17\%$, not +$11\%$.} +\end{table} Two things follow. The wide-bracket gap to Chandrupatla is real and it is \emph{diffuse}: measured per family it is $+57$, $+54$, $+46$, $+43$, $+35$ and smaller, with no single problem costing more than $35$, so it is not the kind of deficit a single mechanism removes. And on that set our shape reading ($2159$ -re-scored) is barely better than bisection ($2136$); only the dissimilarity -reading ($1740$) is clearly ahead. Both statements are reproduced by -\texttt{diag\_rescore\_cha.py} and \texttt{diag\_cha\_gap.py}. +re-scored) is \emph{worse} than bisection ($2136$); only the dissimilarity +reading ($1740$) is clearly ahead. Every statement in this subsection is +reproduced by \texttt{diag\_rescore\_cha.py}, \texttt{diag\_rescore\_aps.py}, +\texttt{diag\_fun7\_root.py} and \texttt{diag\_cha\_gap.py}, whose outputs are +kept beside them. \section{Negative results} diff --git a/diag_rescore_aps.py b/diag_rescore_aps.py new file mode 100644 index 0000000..d331d18 --- /dev/null +++ b/diag_rescore_aps.py @@ -0,0 +1,106 @@ +"""Does the canonical Alefeld-Potra-Shi set carry the same scoring artefact as fun7? + +On the wide-bracket set five problems are scored unsolved for every method although every +method returns a point where f(x) = 0 exactly (diag_rescore_cha.py). Every method also +reports 153 of 154 on the canonical set, so exactly one problem there is unsolved by +everyone — this asks whether that is the same artefact, a different one, or a real failure. + +Both criteria are applied to the SAME runs: + strict : |x - declared_root| <= ACC (the harness rule) + honest : strict OR f(x) == 0 (a root is a root) +and every problem that the two criteria disagree about is printed with its diagnosis. + + python -u diag_rescore_aps.py +""" + +from __future__ import annotations + +import math + +from simroot.baselines import (best_of, bisection, brent, brent_scipy, chandrupatla, + chandrupatla_scipy, itp, toms748) +from simroot.pure import pure_root +from sweep_full import ACC, MAXFEV, XTOL, load_problems + +METHODS = { + "ours (shape)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV, similarity="ratio", + scale="derived3", guard="safe", + prestep="slope", points="shape"), + "ours (dissimilar)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV, similarity="ratio", + scale="derived3", guard="safe", + prestep="slope", direction="dissimilar"), + "ours (published proj)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV, + similarity="ratio", scale="derived3", + guard="project"), + "chandrupatla": lambda f, a, b: best_of(chandrupatla, chandrupatla_scipy)(f, a, b, XTOL, + MAXFEV), + "toms748": lambda f, a, b: toms748(f, a, b, XTOL, MAXFEV), + "brent": lambda f, a, b: best_of(brent, brent_scipy)(f, a, b, XTOL, MAXFEV), + "itp": lambda f, a, b: itp(f, a, b, XTOL, MAXFEV), + "bisection": lambda f, a, b: bisection(f, a, b, XTOL, MAXFEV), +} +TINY = 2.2250738585072014e-308 # smallest normal double, for reporting only + + +def main(): + probs = [p for p in load_problems() if p[0] == "APS"] + print(f"APS problems: {len(probs)}") + print(f"\n{'method':<22} {'strict solved/total':>22} {'honest solved/total':>22} " + f"{'disagreements':>14}") + disagree = {} + never_strict = {pid: 0 for _s, pid, *_ in probs} + for name, run in METHODS.items(): + s_solved = s_total = h_solved = h_total = 0 + diffs = [] + for _s, pid, f, a, b, root in probs: + try: + x, n, conv = run(f, a, b) + except Exception: + x, n, conv = float("nan"), MAXFEV, False + x = float(x) + n = int(n) + fin = conv and math.isfinite(x) + strict = bool(fin and abs(x - root) <= ACC) + honest = bool(strict or (fin and f(x) == 0.0)) + s_total += n if strict else MAXFEV + h_total += n if honest else MAXFEV + s_solved += int(strict) + h_solved += int(honest) + never_strict[pid] += int(strict) + if honest != strict: + diffs.append((pid, x, root, f(x), n)) + disagree[name] = diffs + print(f"{name:<22} {s_solved:8d}/154 {s_total:8d} {h_solved:8d}/154 {h_total:8d} " + f"{len(diffs):14d}") + + print("\nproblems where the two criteria disagree (a root found, scored a failure):") + any_diff = False + for name, diffs in disagree.items(): + for pid, x, root, fx, n in diffs: + any_diff = True + print(f" {name:<22} {pid:<12} x={x:+.10g} root={root:+.10g} " + f"f(x)={fx:+.3e} after {n} evals") + if not any_diff: + print(" none — the canonical set does NOT carry the fun7 artefact") + + print("\nproblems no method solved under the strict rule:") + for pid, cnt in sorted(never_strict.items(), key=lambda kv: kv[1])[:6]: + print(f" {pid:<12} solved by {cnt} of {len(METHODS)} methods") + + # inspect the hardest one in the same way fun7 was inspected + worst = min(never_strict, key=lambda k: never_strict[k]) + for _s, pid, f, a, b, root in probs: + if pid != worst: + continue + print(f"\ndiagnosis of {pid}: bracket [{a:.6g}, {b:.6g}] declared root {root:.10g}") + print(f" f(a)={f(a):+.6e} f(b)={f(b):+.6e} f(root)={f(root):+.6e}") + for t in (1e-12, 1e-9, 1e-6, 1e-3): + print(f" f(root+{t:g}) = {f(root + t):+.6e} f(root-{t:g}) = {f(root - t):+.6e}") + x, n, conv = METHODS["ours (dissimilar)"](f, a, b) + print(f" ours returns x={float(x):+.12g} after {n} evals, f(x)={f(float(x)):+.3e}, " + f"|x-root|={abs(float(x) - root):.3g}") + print(f" |f(x)| below smallest normal? {abs(f(float(x))) < TINY}") + + +if __name__ == "__main__": + main() diff --git a/results_rescore_aps.txt b/results_rescore_aps.txt new file mode 100644 index 0000000..19a277f --- /dev/null +++ b/results_rescore_aps.txt @@ -0,0 +1,38 @@ +APS problems: 154 + +method strict solved/total honest solved/total disagreements +ours (shape) 153/154 2527 154/154 2337 1 +ours (dissimilar) 153/154 3123 154/154 2934 1 +ours (published proj) 153/154 2806 154/154 2634 1 +chandrupatla 153/154 2785 154/154 2593 1 +toms748 153/154 2820 154/154 2636 1 +brent 153/154 2889 154/154 2707 1 +itp 153/154 3957 154/154 3767 1 +bisection 153/154 7530 154/154 7338 1 + +problems where the two criteria disagree (a root found, scored a failure): + ours (shape) aps.13.00 x=+0.008638489218 root=+0 f(x)=+0.000e+00 after 10 evals + ours (dissimilar) aps.13.00 x=+0.0248491106 root=+0 f(x)=+0.000e+00 after 11 evals + ours (published proj) aps.13.00 x=-0.01020453457 root=+0 f(x)=+0.000e+00 after 28 evals + chandrupatla aps.13.00 x=-0.008282706313 root=+0 f(x)=+0.000e+00 after 8 evals + toms748 aps.13.00 x=-0.0002236853352 root=+0 f(x)=+0.000e+00 after 16 evals + brent aps.13.00 x=+0.0154213326 root=+0 f(x)=+0.000e+00 after 18 evals + itp aps.13.00 x=+0.03079215104 root=+0 f(x)=+0.000e+00 after 10 evals + bisection aps.13.00 x=+0.015625 root=+0 f(x)=+0.000e+00 after 8 evals + +problems no method solved under the strict rule: + aps.13.00 solved by 0 of 8 methods + aps.01.00 solved by 8 of 8 methods + aps.02.00 solved by 8 of 8 methods + aps.02.01 solved by 8 of 8 methods + aps.02.02 solved by 8 of 8 methods + aps.02.03 solved by 8 of 8 methods + +diagnosis of aps.13.00: bracket [-1, 4] declared root 0 + f(a)=-3.678794e-01 f(b)=+3.757652e+00 f(root)=+0.000000e+00 + f(root+1e-12) = +0.000000e+00 f(root-1e-12) = +0.000000e+00 + f(root+1e-09) = +0.000000e+00 f(root-1e-09) = +0.000000e+00 + f(root+1e-06) = +0.000000e+00 f(root-1e-06) = +0.000000e+00 + f(root+0.001) = +0.000000e+00 f(root-0.001) = +0.000000e+00 + ours returns x=+0.0248491106033 after 11 evals, f(x)=+0.000e+00, |x-root|=0.0248 + |f(x)| below smallest normal? True