vvs/sim-numerics
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

the scoring artefact is in BOTH sets: aps.13.00 joins the five fun7 problems — 6 of 199 problems where f underflows to exactly zero on a ~3.2e-2 neighbourhood of the declared root, so every method returns a genuine root (|f(x)| below the smallest normal) and every method is scored a failure. About 1150 evaluations of EVERY two-set total are this artefact; re-scored, all methods solve 199/199 except Brent 197. Ranking unchanged: ours-by-shape 2337 on APS against chandrupatla 2593, chandrupatla 1487 on CHA against our 1740. Written into the paper as a table with both criteria; pdflatex clean, verifier green

Browse Source 
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
vvs
2026-09-21 00:01:35 +01:00
co-authored by Claude Opus 5
parent 6b5870f815
commit 8d20816163
4 changed files with 190 additions and 22 deletions
Download Patch File Download Diff File Expand all files Collapse all files
article/sim_numerics.pdf
BIN
View File
Binary file not shown.
article/sim_numerics.tex
+46 -22
View File
@@ -563,37 +563,61 @@ The readings do not compose. Taken together, (2) and (3) give $2884$ and
$2859$ --- worse than either alone. Two selections that are each an improvement
interfere, which is a reason to measure composition rather than assume it.
\subsection{Five problems of the wide-bracket set are scored unsolved for every
method}
\subsection{Six problems of the two sets are scored unsolved for every method}
\label{sec:fun7}
Chasing the remaining gap to Chandrupatla on the wide-bracket set led to a
property of the set itself. On the five problems of its \texttt{fun7} family the
declared root is $0$, but $f$ vanishes \emph{identically} on a neighbourhood of
it: $f(0+t)=0$ for every $t$ up to at least $3.2\times10^{-2}$. Every method
therefore stops after $8$ to $22$ evaluations at a point where $f(x)=0$
\emph{exactly} --- a genuine root --- and every method is then scored as having
failed, because the harness asks for $|x-\text{root}|\le10^{-6}$ and the
returned point sits up to $2.8\times10^{-2}$ away.
Chasing the remaining gap to Chandrupatla led to a property of the benchmarks
themselves. On the five problems of the wide-bracket family \texttt{fun7}, and on
the single canonical problem \texttt{aps.13.00}, the declared root is $0$ while
$f$ vanishes \emph{identically} on a neighbourhood of it in floating point:
$f(0\pm t)=0$ for every $t$ up to at least $3.2\times10^{-2}$, because the
exponential factor underflows --- at the returned points $|f(x)|$ is below the
smallest normal double. Every method therefore stops after $8$ to $28$
evaluations at a point where $f(x)=0$ \emph{exactly} --- a genuine root --- and
every method is then scored as having failed, because the harness asks for
$|x-\text{root}|\le10^{-6}$ while the returned point sits up to
$2.8\times10^{-2}$ away.
The consequence is arithmetic. Each such problem charges the full budget of
$200$ evaluations to every method, so $1000$ evaluations of every wide-bracket
total --- ours, Chandrupatla's, ITP's, bisection's --- are this artefact, and the
``$40$ of $45$ solved'' that every method reports is a statement about the
scoring rather than about the methods. Re-scored with the criterion ``a root is a
root'' ($|x-\text{root}|\le10^{-6}$ \emph{or} $f(x)=0$), the same runs give
$45/45$ for every method except Brent's $43/45$, and the totals fall to
Chandrupatla $1487$, ours $1740$, ITP $1907$, bisection $2136$, TOMS\,748
$3088$. The ranking does not change; the magnitudes do, and with them the size
of the remaining gap: $253$ evaluations, or $17\%$, not $11\%$.
$200$ evaluations to every method: about $1150$ evaluations of every reported
two-set total --- ours, Chandrupatla's, TOMS\,748's, ITP's, Brent's, bisection's
--- are this artefact, and the ``$153$ of $154$'' and ``$40$ of $45$'' that every
method reports are statements about the scoring rather than about the methods.
Re-scored with the criterion ``a root is a root''
($|x-\text{root}|\le10^{-6}$ \emph{or} $f(x)=0$), the same runs solve $199$ of
$199$ for every method except Brent's $197$, and the totals fall as follows.
\begin{table}[h]
\centering
\begin{tabular}{lrrrr}
\hline
& \multicolumn{2}{c}{APS-154} & \multicolumn{2}{c}{CHA-45} \\
method & as scored & re-scored & as scored & re-scored \\
\hline
ours, points by shape & $\mathbf{2527}$ & $\mathbf{2337}$ & $3108$ & $2159$ \\
ours, projection variant & $2806$ & $2634$ & $3220$ & $2357$ \\
Chandrupatla & $2785$ & $2593$ & $\mathbf{2423}$ & $\mathbf{1487}$ \\
TOMS\,748 & $2820$ & $2636$ & $4000$ & $3088$ \\
Brent & $2889$ & $2707$ & $4099$ & $3222$ \\
ITP & $3957$ & $3767$ & $2857$ & $1907$ \\
bisection & $7530$ & $7338$ & $3090$ & $2136$ \\
\hline
\end{tabular}
\caption{The same runs under the harness criterion and under ``a root is a
root''. The ranking is unchanged; the magnitudes are not, and neither is the size
of the remaining gap on the wide-bracket set: $253$ evaluations, or $17\%$, not
$11\%$.}
\end{table}
Two things follow. The wide-bracket gap to Chandrupatla is real and it is
\emph{diffuse}: measured per family it is $+57$, $+54$, $+46$, $+43$, $+35$ and
smaller, with no single problem costing more than $35$, so it is not the kind of
deficit a single mechanism removes. And on that set our shape reading ($2159$
re-scored) is barely better than bisection ($2136$); only the dissimilarity
reading ($1740$) is clearly ahead. Both statements are reproduced by
\texttt{diag\_rescore\_cha.py} and \texttt{diag\_cha\_gap.py}.
re-scored) is \emph{worse} than bisection ($2136$); only the dissimilarity
reading ($1740$) is clearly ahead. Every statement in this subsection is
reproduced by \texttt{diag\_rescore\_cha.py}, \texttt{diag\_rescore\_aps.py},
\texttt{diag\_fun7\_root.py} and \texttt{diag\_cha\_gap.py}, whose outputs are
kept beside them.
\section{Negative results}
diag_rescore_aps.py
+106
View File
@@ -0,0 +1,106 @@
"""Does the canonical Alefeld-Potra-Shi set carry the same scoring artefact as fun7?
On the wide-bracket set five problems are scored unsolved for every method although every
method returns a point where f(x) = 0 exactly (diag_rescore_cha.py). Every method also
reports 153 of 154 on the canonical set, so exactly one problem there is unsolved by
everyone — this asks whether that is the same artefact, a different one, or a real failure.
Both criteria are applied to the SAME runs:
strict : |x - declared_root| <= ACC (the harness rule)
honest : strict OR f(x) == 0 (a root is a root)
and every problem that the two criteria disagree about is printed with its diagnosis.
python -u diag_rescore_aps.py
"""
from __future__ import annotations
import math
from simroot.baselines import (best_of, bisection, brent, brent_scipy, chandrupatla,
chandrupatla_scipy, itp, toms748)
from simroot.pure import pure_root
from sweep_full import ACC, MAXFEV, XTOL, load_problems
METHODS = {
"ours (shape)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV, similarity="ratio",
scale="derived3", guard="safe",
prestep="slope", points="shape"),
"ours (dissimilar)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV, similarity="ratio",
scale="derived3", guard="safe",
prestep="slope", direction="dissimilar"),
"ours (published proj)": lambda f, a, b: pure_root(f, a, b, XTOL, MAXFEV,
similarity="ratio", scale="derived3",
guard="project"),
"chandrupatla": lambda f, a, b: best_of(chandrupatla, chandrupatla_scipy)(f, a, b, XTOL,
MAXFEV),
"toms748": lambda f, a, b: toms748(f, a, b, XTOL, MAXFEV),
"brent": lambda f, a, b: best_of(brent, brent_scipy)(f, a, b, XTOL, MAXFEV),
"itp": lambda f, a, b: itp(f, a, b, XTOL, MAXFEV),
"bisection": lambda f, a, b: bisection(f, a, b, XTOL, MAXFEV),
}
TINY = 2.2250738585072014e-308 # smallest normal double, for reporting only
def main():
probs = [p for p in load_problems() if p[0] == "APS"]
print(f"APS problems: {len(probs)}")
print(f"\n{'method':<22} {'strict solved/total':>22} {'honest solved/total':>22} "
f"{'disagreements':>14}")
disagree = {}
never_strict = {pid: 0 for _s, pid, *_ in probs}
for name, run in METHODS.items():
s_solved = s_total = h_solved = h_total = 0
diffs = []
for _s, pid, f, a, b, root in probs:
try:
x, n, conv = run(f, a, b)
except Exception:
x, n, conv = float("nan"), MAXFEV, False
x = float(x)
n = int(n)
fin = conv and math.isfinite(x)
strict = bool(fin and abs(x - root) <= ACC)
honest = bool(strict or (fin and f(x) == 0.0))
s_total += n if strict else MAXFEV
h_total += n if honest else MAXFEV
s_solved += int(strict)
h_solved += int(honest)
never_strict[pid] += int(strict)
if honest != strict:
diffs.append((pid, x, root, f(x), n))
disagree[name] = diffs
print(f"{name:<22} {s_solved:8d}/154 {s_total:8d} {h_solved:8d}/154 {h_total:8d} "
f"{len(diffs):14d}")
print("\nproblems where the two criteria disagree (a root found, scored a failure):")
any_diff = False
for name, diffs in disagree.items():
for pid, x, root, fx, n in diffs:
any_diff = True
print(f" {name:<22} {pid:<12} x={x:+.10g} root={root:+.10g} "
f"f(x)={fx:+.3e} after {n} evals")
if not any_diff:
print(" none — the canonical set does NOT carry the fun7 artefact")
print("\nproblems no method solved under the strict rule:")
for pid, cnt in sorted(never_strict.items(), key=lambda kv: kv[1])[:6]:
print(f" {pid:<12} solved by {cnt} of {len(METHODS)} methods")
# inspect the hardest one in the same way fun7 was inspected
worst = min(never_strict, key=lambda k: never_strict[k])
for _s, pid, f, a, b, root in probs:
if pid != worst:
continue
print(f"\ndiagnosis of {pid}: bracket [{a:.6g}, {b:.6g}] declared root {root:.10g}")
print(f" f(a)={f(a):+.6e} f(b)={f(b):+.6e} f(root)={f(root):+.6e}")
for t in (1e-12, 1e-9, 1e-6, 1e-3):
print(f" f(root+{t:g}) = {f(root + t):+.6e} f(root-{t:g}) = {f(root - t):+.6e}")
x, n, conv = METHODS["ours (dissimilar)"](f, a, b)
print(f" ours returns x={float(x):+.12g} after {n} evals, f(x)={f(float(x)):+.3e}, "
f"|x-root|={abs(float(x) - root):.3g}")
print(f" |f(x)| below smallest normal? {abs(f(float(x))) < TINY}")
if __name__ == "__main__":
main()
results_rescore_aps.txt
+38
View File
@@ -0,0 +1,38 @@
APS problems: 154
method strict solved/total honest solved/total disagreements
ours (shape) 153/154 2527 154/154 2337 1
ours (dissimilar) 153/154 3123 154/154 2934 1
ours (published proj) 153/154 2806 154/154 2634 1
chandrupatla 153/154 2785 154/154 2593 1
toms748 153/154 2820 154/154 2636 1
brent 153/154 2889 154/154 2707 1
itp 153/154 3957 154/154 3767 1
bisection 153/154 7530 154/154 7338 1
problems where the two criteria disagree (a root found, scored a failure):
ours (shape) aps.13.00 x=+0.008638489218 root=+0 f(x)=+0.000e+00 after 10 evals
ours (dissimilar) aps.13.00 x=+0.0248491106 root=+0 f(x)=+0.000e+00 after 11 evals
ours (published proj) aps.13.00 x=-0.01020453457 root=+0 f(x)=+0.000e+00 after 28 evals
chandrupatla aps.13.00 x=-0.008282706313 root=+0 f(x)=+0.000e+00 after 8 evals
toms748 aps.13.00 x=-0.0002236853352 root=+0 f(x)=+0.000e+00 after 16 evals
brent aps.13.00 x=+0.0154213326 root=+0 f(x)=+0.000e+00 after 18 evals
itp aps.13.00 x=+0.03079215104 root=+0 f(x)=+0.000e+00 after 10 evals
bisection aps.13.00 x=+0.015625 root=+0 f(x)=+0.000e+00 after 8 evals
problems no method solved under the strict rule:
aps.13.00 solved by 0 of 8 methods
aps.01.00 solved by 8 of 8 methods
aps.02.00 solved by 8 of 8 methods
aps.02.01 solved by 8 of 8 methods
aps.02.02 solved by 8 of 8 methods
aps.02.03 solved by 8 of 8 methods
diagnosis of aps.13.00: bracket [-1, 4] declared root 0
f(a)=-3.678794e-01 f(b)=+3.757652e+00 f(root)=+0.000000e+00
f(root+1e-12) = +0.000000e+00 f(root-1e-12) = +0.000000e+00
f(root+1e-09) = +0.000000e+00 f(root-1e-09) = +0.000000e+00
f(root+1e-06) = +0.000000e+00 f(root-1e-06) = +0.000000e+00
f(root+0.001) = +0.000000e+00 f(root-0.001) = +0.000000e+00
ours returns x=+0.0248491106033 after 11 evals, f(x)=+0.000e+00, |x-root|=0.0248
|f(x)| below smallest normal? True