Source

engines/design23_v1/notes/v_and_beta_gradients_are_untrustworthy.md · assembled 2026-09-05 15:14 UTC.

---

The V and beta gradients are secants, and they cannot be trusted

Opened 2026-08-08, after the composite Q/V/beta campaign stalled at accepted step 7 with six consecutive line-search failures and no accepted step in six hours of wall clock.

The headline is in the name of the thing. _objective_gradient (campaigns/composite_qv_beta_40holes/run_pure_bfgs.py:174) assembles

gradient = -(cavity_weight * (dlogQ - dlogV) + dlogbeta)

where only dlogQ is a gradient. dlogV and dlogbeta come from _minimum_norm_secant, which fits a linear response to the history of accepted steps. The code says so plainly -- "V and beta derivatives become data-informed secants after the first accepted direction, while dlogQ is exact at every iterate" -- and the campaign ran for seven accepted steps before the consequence surfaced.

What the stall measured

Iteration 15's line search halved its step ten times, from 0.078 nm down to 0.15 pm, and recorded Q, V and beta at every attempt. That is a ten-point directional derivative measurement along the failing direction, and its alpha -> 0 limits are flat to four digits across a 64x range of step size. Against the three predicted terms:

term	predicted	measured	ratio	source	share of predicted slope
dlogQ/dalpha	+8.555e-5	-7.624e-4	-0.11	exact	0.1%
dlogV/dalpha	-1.965e-2	+3.216e-3	-6.11	secant	16.2%
dlogbeta/dalpha	+4.880e-2	+3.792e-4	+128.7	secant	83.7%
dJ/dalpha	+5.831e-2	-1.538e-3	-37.9

The decomposition is exact rather than approximate: rebuilding the directional derivative from the three terms returns -5.831261e-02 against the recorded -5.831261e-02, digit for digit. So the search direction at step 7 was 83.7% determined by a quantity that overestimates the true response by two orders of magnitude, and the objective's one honest gradient contributed 0.1%.

Why the secant blows up

_minimum_norm_secant solves a least-norm problem over at most sixteen past accepted steps, regularized by ridge = 1e-8 * mean(diag(gram)). At step 7 it had seven steps in a 60-parameter space, and their singular values are

1.702, 0.727, 0.610, 0.599, 8.394e-6, 1.285e-9, 7.606e-13

so the numerical rank is four. The last three accepted steps were very nearly parallel to earlier ones, which is not a malfunction -- it is what a converging BFGS does. The shipped ridge of 1.0e-8 sits above the fifth mode's gram eigenvalue of 7.046e-11 but nowhere near far enough above it: that mode ends up weighted roughly 1400x more per unit response than the dominant one. A trace of noise along a nearly-degenerate step direction becomes the largest term in the gradient.

Re-solving with a truncated SVD instead, which needs no physics at all:

relative cut	rank	beta ratio	V ratio	composite dJ/dalpha ratio
shipped (ridge)	7	128.69	-6.11	-37.91
1e-6	5	18130.81	-797.28	-5273.54
1e-4 ... 1e-1	4	1.84	-0.54	-1.02

Admitting the fifth direction is catastrophic and truncating below it is stable at any threshold between 1e-4 and 1e-1. That is the mechanism confirmed from both sides.

The part a better ridge does not fix

At rank four the predicted composite slope is +1.570e-3 against a measured -1.538e-3. Right magnitude, still the wrong sign. Repairing the conditioning alone would have bought confidently sized steps in the wrong direction rather than absurdly sized ones.

Only beta has the correct sign after truncation. V is -0.54 and Q is -0.11, and the Q one is worth its own line: the exact analytical gradient says Q rises along this direction while measurement says it falls, consistently at every step size sampled. It contributed 0.1% of this direction so it did not cause this stall, but "exact at every iterate" is carrying less weight than the name implies, and it has not been checked against measurement anywhere else in this campaign either.

What follows

V and beta must not steer an optimizer in this engine until they have real gradients. The secants are not a cheap approximation to a derivative; they are a fit whose conditioning degrades exactly as the optimizer converges, so the failure is structural rather than a tuning accident. Optimizing Q alone, with V and beta demoted to measured acceptance constraints, is the only configuration currently supported by evidence.

Three repairs, none of them yet done:

1. Truncate the secant below its rank and add a gradient-versus-measurement gate to the line search. The gate would have caught this at iteration 10 rather than at iteration 15. It does not fix the sign.
2. Give V and beta real gradients. The meep lab has a derived and partially validated dV/dh adjoint at ratio 0.72 with its discrepancy localized to the boundary kernel's cross term; see the mode-volume note.
3. Check dlogQ against measurement in its own right, at more than one iterate. Nothing in this campaign has ever done so.

One process defect worth fixing regardless: after iteration 10 reset the BFGS memory, iterations 11-15 are bit-identical -- same direction, same directional derivative -0.05831260757897166, same ten alphas, same ten failures -- because steepest descent from an unchanged incumbent is deterministic. Four of the six hours produced no new information. The stall detector fires on six consecutive failures; it should fire on the second identical direction.

Reproducing

All three stages read the stalled run's state.json and need no new physics except the exact dlogQ, which costs one confirmation pole solve of about 350 s:

engines/design23_v1/runs/20260728T013605Z_design23_40hole_composite_bfgs/state.json

The measured slopes are in history[-1].line_search, the secants are a pure function of history, and the incumbent reproduces on this host to 6.7e-10 after the /root to /home/qluster migration.