Source

Engine design23_v1 · package package/design23_recentered_optimizer · assembled 2026-07-29 15:57 UTC.

Repaired optimizer: first Tidy3D gradient gate¶

Outcome¶

The pole/branch-subtracted Design23 solver is now the optimizer objective for all 126 hole coordinates. Its reverse-mode gradient is numerically correct, and its first finite trust-region step is large enough for an FDTD sign test.

This is not yet permission to run a long optimization. The next paid action is the three-design Tidy3D gate below.

Gradient validation¶

At the 412-node development grid, the saved iteration-0-to-30 direction was scaled to a 0.1 nm maximum coordinate displacement. The fixed-frequency tracked eigenvalue derivative was

central finite difference  -9.9476613141e-5 + 6.1691679430e-8 i
analytic VJP               -9.9476626041e-5 + 6.1700312856e-8 i

The corresponding Q derivatives were

central finite difference   3.0939549116
analytic VJP                3.0939341486
relative error              6.71e-6

The reverse pass differentiates the finite-hole Fourier projections in the smooth real-axis continuum and every ordered real-space strip-pole residue. It does not differentiate the rejected legacy contour operator.

Safeguarded first step¶

The full repaired Q gradient was normalized so its largest absolute coordinate change is one nanometre per unit step. Width and thickness were frozen. The line search tracked one branch and required overlap at least 0.999.

The final 812-node evaluation gives:

case	max coordinate change	predicted Q	Q/Q0	wavelength (nm)	branch overlap
minus_0p5nm	-0.50 nm	5492.77	0.75551	796.6453	0.999614
minus_0p25nm	-0.25 nm	6348.88	0.87326	796.4686	0.999905
baseline	0	7270.34	1.00000	796.2880	1.000000
plus_0p25nm	+0.25 nm	8189.34	1.12640	796.1034	0.999907
plus_0p5nm	+0.50 nm	8997.81	1.23761	795.9150	0.999632

A +1 nm probe predicted a larger Q, but its branch overlap fell below 0.999 and it was excluded from the FDTD batch.

What to run¶

Activate the same environment used for the earlier Design23 Tidy3D jobs, then estimate the decisive three-job batch:

$env:MPLBACKEND = "Agg"
$env:OMP_NUM_THREADS = "1"
$env:OPENBLAS_NUM_THREADS = "1"
$env:MKL_NUM_THREADS = "1"
$env:NUMEXPR_NUM_THREADS = "1"

python .\run_tidy3d_validation_batch.py .\tidy3d_validation_batch.json `
  --output .\tidy3d_gradient_gate `
  --labels baseline minus_0p5nm plus_0p5nm `
  --estimate-only

Inspect tidy3d_gradient_gate/cost_estimates.json, then submit exactly the same three labels:

python .\run_tidy3d_validation_batch.py .\tidy3d_validation_batch.json `
  --output .\tidy3d_gradient_gate `
  --labels baseline minus_0p5nm plus_0p5nm `
  --submit

The runner uses 48 ps, 14 steps per wavelength, 12 PML layers, the exact air/SiN/anthracene/PVA/substrate stack, and (1,-1,0) symmetry. It writes the resonance tables, field diagnostics, and gradient_gate_results.json.

Acceptance decision¶

Primary sign gate:

Q_FDTD(plus_0p5nm) > Q_FDTD(minus_0p5nm)

Improvement gate:

Q_FDTD(plus_0p5nm) > Q_FDTD(baseline)

The first condition validates the gradient direction even if an absolute offset or weak nonlinearity remains. The second is required before accepting the step into an optimization trajectory.

If both pass, run the smaller symmetric pair without rerunning cached data:

python .\run_tidy3d_validation_batch.py .\tidy3d_validation_batch.json `
  --output .\tidy3d_gradient_gate `
  --labels minus_0p25nm plus_0p25nm `
  --submit

The full gate passes if both positive cases beat their equal-magnitude negative controls and at least one positive case beats the baseline. Compare normalized Q ratios; do not calibrate the analytical objective to the baseline FDTD Q.

Local verification status¶

All 21 offline regressions pass. The current execution environment lacks the tidy3d Python package, so simulation construction was not instantiated here; the supplied runner uses the same fdtd_core.py geometry, resonance fitting, and diagnostics already used for the previous Design23 checkpoints.