Device23 Q optimizer trust rebuild

Open the live physical-pattern dashboard Open the live surrogate-only continuation

Result

The previous free-form trajectory is rejected as a physical Q optimization. It did not reach a noise floor or a Q stationary point. On matched canonical geometries, its persistent-mesh FEM Q increased from 2047.77 at step 2 to 2521.30 at step 10, while qualified Tidy3D Q decreased monotonically from 923.53 to 822.88. The changes are +23.12% and −10.90%, respectively.

Each Tidy3D point uses the same level-set-to-polygon conversion, 740--800 nm source window, 16 ps maximum runtime, 14 points per wavelength, automesh with no override regions, symmetry (1, -1, 0), and identical resonance-analysis protocol. Fits were repeated over three central monitors and three trace windows. All five points passed the stability gate; the reported Q uncertainty floor is 2%. The four new jobs cost 0.3995 FC in total. Including the earlier step-10 job, those retrospective checks spent 0.5030 FC.

The replacement optimizer has now produced its first resolved physical gain. For the smooth uniform-hole coordinate at ±0.5 nm RMS boundary motion, qualified Tidy3D returned Q = 892.73 for the negative sign and Q = 1001.35 for the positive sign, against the qualified rollback incumbent Q = 923.53. The positive candidate's 2%-lower bound (981.32) is above the incumbent's 2%-upper bound (942.00), so the +8.43% gain was promoted. The two paired jobs cost 0.2111 FC; at that promotion, total shared-ledger spend was 0.7142 / 5.0 FC. Subsequent live-search evaluations are reflected on the dashboard and in fdtd_budget.json.

Left: qualified Tidy3D Q for the incumbent and both signs; error bars are the conservative 2% analysis floor. Right: the central log-Q slope from the two fresh-remesh FEM poles and the two matched Tidy3D poles. The sign transfers for this smooth coordinate, although FEM absolute Q does not. Sources: fdtd_step_000002_diagnostic, fdtd_step_012000_diagnostic, fdtd_step_012001_diagnostic, and independent_remesh_scan.json; generator: plot_physical_pattern_bootstrap.py.

Matched-axis complete-material level sets before and after the accepted +0.5 nm uniform-hole motion. Blue is Si₃N₄ and white is air; the change panel shows removed dielectric in red and added dielectric in green. Sources: rollback field_step_000002.npz and promoted hole_uniform_normal__0.5nm__plus.npz; 4 nm level-set grid; hashes and axes are in physical_bootstrap_manifest.json.

Accepted candidate at 770.423 nm and qualified Q = 1001.35. Quantity: Hann-projected vector |E(ω₀)|² on the XY midplane; per-panel robust color normalization. Source: retained Tidy3D HDF5 from diagnostic step 12001; the time window, monitor, frequency, and plotted bounds are recorded in physical_bootstrap_manifest.json.

The first supervised 0.5 nm sweep then found another resolved improvement. The localized hole_defect_gaussian coordinate returned Q = 945.08 and Q = 1051.73; its positive sign cleared the incumbent's uncertainty interval and was promoted. A later +1.25 nm defect-hole step produced the next formal paired checkpoint, Q = 1115.17 at 767.910 nm. That is +20.75% from the physical rollback baseline.

The final affordable sign of the following sweep was even better. The −1.5625 nm hole-ellipticity candidate reached qualified Q = 1183.52 at 768.555 nm. Its 2%-lower bound is 1159.85, above the Q = 1115.17 checkpoint's 2%-upper bound of 1137.47; all nine monitor/window fits agreed. The opposite sign could not be started within the remaining budget, so this geometry is retained as the physically qualified seed of the offline branch, but is not relabeled as a completed paired-search checkpoint. Relative to the original Q = 923.53 rollback geometry, it is a +28.15% physical gain.

Matched-axis rollback geometry and best qualified physical candidate used by the offline continuation. Blue is Si₃N₄, white is air, red marks dielectric removed, and green marks dielectric added relative to the rollback geometry. Sources: rollback field_step_000002.npz and hole_defect_ellipticity__1.5625nm__minus.npz; hashes, material colors, and axes are in physical_bootstrap_manifest.json.

Best qualified candidate at 768.555 nm and Q = 1183.52. Quantity: Hann-projected vector |E(ω₀)|² on the XY midplane; robust per-panel color normalization. Source: retained diagnostic-step-20035 Tidy3D HDF5; monitor, time window, bounds, and frequency are in physical_bootstrap_manifest.json.

Top: every completed qualified Tidy3D candidate, with the conservative 2% error floor and the incumbent active at that evaluation; orange points are sidewall controls and blue points are hole controls. Bottom: paired central physical log-Q slopes at the tested 0.5--1.5625 nm RMS coordinate amplitudes. The final unpaired blue point is the Q = 1183.52 offline seed. Source: live state.json; generator: plot_physical_pattern_history.py; exact values and color choices are in physical_pattern_history_manifest.json.

Actual simulation stack: 600 nm SiO₂ substrate, 300 nm etched Si₃N₄, 200 nm unetched anthracene, 200 nm unetched PVA, then air. Layer indices are shown in the figure. Source: device23_40hole_fdtd_seed.json; generator: plot_geometry.py.

Left: frozen-connectivity FEM Q and independently qualified Tidy3D Q versus accepted boundary step. Right: cumulative FEM and Tidy3D log-Q change from the same step-2 boundary. Sources: optimizer_history.jsonl and fdtd_step_00000{2,4,6,8,10}_diagnostic/diagnostic_result.json. Tidy3D points are median fits across three monitors and three time windows. Generator: plot_trust_rebuild.py; exact source paths and values are recorded in docs/assets/device23_q_trust_rebuild/figure_manifest.json.

What failed

The additional sidewall freedom was real but was not meaningfully explored. By step 10 the sidewall modulation was only 2.91 nm peak-to-peak, comparable to the 4 nm level-set grid, while the maximum hole-contour displacement from the persistent reference was about 4.9 nm. The optimizer stopped because its absolute VectorH1 ALE transfer error reached the 30% gate. A trial of only 0.05 nm inherited that cumulative error, so step reduction could never pass.

More importantly, common-connectivity derivative agreement was not a physical gradient certificate. It showed that the boundary integral differentiated the same discrete deformed operator consistently; it did not show that a freshly meshed geometry, or FDTD, followed that direction. Q depends on the small imaginary part of the pole, so a mesh change that is negligible for wavelength can dominate the inferred loss derivative.

Step-2 and step-10 complete-material level sets at matched axes with an explicit Si₃N₄/air legend. The lower panel magnifies the signed material change: red removes Si₃N₄ and green adds it. Sources: field_step_000002.npz and field_step_000010.npz; 4 nm level-set grid; generator and hashes in figure_manifest.json.

The tracked field remains the same localized cavity family, so the disagreement is not explained by an obvious mode switch.

Step-2 Tidy3D XY-midplane resonant intensity at 770.792 nm and qualified Q = 923.53. Quantity: Hann-projected vector |E(ω₀)|²; robust per-panel color normalization. Source: retained step-2 Tidy3D HDF5; monitor and temporal window are recorded in figure_manifest.json.

Step-10 Tidy3D XY-midplane resonant intensity at 769.834 nm and qualified Q = 822.88. The component, slice, temporal projection, and normalization match the step-2 panel, although each panel has its own robust color scale. Source: retained step-10 Tidy3D HDF5.

Implemented replacement

The stopped trajectory is preserved, and step 2 is the physical rollback baseline. The replacement infrastructure now enforces:

• A canonical contour-and-stack snapshot and SHA-256 identity shared by FEM and Tidy3D.
• Bounded mesh epochs. The active geometry becomes a new zero-deformation reference before 10% maximum or 2% RMS cumulative ALE transfer error. A zero-size step therefore has zero incremental error.
• Field-overlap-first mode continuation, with frequency and localization as secondary checks.
• An arclength Sobolev metric with 24 nm hole and 80 nm sidewall length scales. Hole and sidewall blocks are RMS-scaled separately, and no localized gradient hotspot may exceed three times the block RMS motion.
• A macro Tidy3D trust gate. A statistically resolved Q decrease rolls back to the last verified checkpoint, shrinks the trust radius, and resets the mesh epoch. An unresolved ringdown pauses instead of silently steering the run.
• A resumable independent-remesh certificate over five distributed hole controls and three sidewall controls. Central differences use exact canonical ±0.5 nm candidates on separately rebuilt FEM meshes.

The first regularized analytic proposal illustrates why the independent gate is mandatory. On the frozen step-2 connectivity it predicted Δlog(Q) = +0.08587 and measured +0.07454, with field overlap 0.9994. The identical canonical candidate on a fresh mesh instead returned Q = 1836.06, or Δlog(Q) = −0.10913. It was rejected before FDTD.

The independent-remesh scan was stopped after its first complete pair. A mere ±0.5 nm uniform-hole perturbation changed the same-fidelity HCurl space from 1,072,508 to 977,001 unknowns, a 9.3% spread. Every hole was resampled to the same 48-node spline and the sidewall to the same 512-node spline, so the jump comes from boundary-conforming tetrahedralization branching, not a contour topology change. The associated FEM central difference therefore cannot rank physical steps by itself. The rejected evidence is preserved in independent_remesh_scan.json and MESH_REJECTION.json.

The optimizer now uses eight smooth coordinates as a trust-building basis: five distributed hole controls and three sidewall controls. It tests both signs with qualified Tidy3D and promotes a geometry only when its lower Q bound exceeds the incumbent's upper bound. Sidewall cosine taper, defect bulge, and uniform width are tried first. The QNM derivative may reorder coordinates only when its field hash matches the current geometry; it has no acceptance authority. L-BFGS remains disabled until multiple physical central differences certify a stable derivative map.

The physical pattern service stopped fail-closed at 4.6270 / 5.0 FC. Its remaining 0.3730 FC was below the next job's 0.4924 FC maximum estimate, so no over-budget task was submitted. The formal paired-search checkpoint is Q = 1115.17; the best individually qualified candidate is Q = 1183.52.

The systemd-supervised continuation now runs locally from that Q = 1183.52 geometry as cavitygrad-device23-surrogate-only-q.service. Its execution path sets fdtd_enabled=false, has a zero-FC ledger, and schedules no Tidy3D diagnostics regardless of accepted-step or boundary-motion counters. It first replays the qualified geometry as an ALE deformation of the known-tractable 1,009,435-DOF frozen reference mesh, then continuously performs Sobolev pure-log-Q hole-and-sidewall ascent. The qualified seed field and geometry are immutable; all later Q values and geometries are explicitly labeled surrogate-only and physically unverified until a future FDTD allowance is provided. A STOP file or stopping that service halts it cleanly.

Corrected offline mode handoff and continuous ascent

The initial offline launch was not healthy. It restarted 220 times because it searched a 0.005 µm⁻¹ window around the qualified Tidy3D pole 1.301143 - 0.0005497i µm⁻¹, while the localized high-Q branch on the frozen FEM reference connectivity was near 1.328658 - 0.0003437i µm⁻¹. Frequency proximity alone therefore rejected the intended branch before optimization.

The corrected initializer first solves the exact frozen-reference FEM mode, retains its full complex H(curl) coefficient vector, deforms the same connectivity to the qualified physical geometry, and ranks candidates by the complex mass overlap with that vector. It recovered the reference mode at Q = 2047.77, then selected the physical-seed geometry's FEM mode at Q = 1932.85, 752.639 nm, with 0.98256 overlap. The canonical geometry hash matches the qualified Tidy3D seed exactly. Initialization now exits fail-closed and systemd does not restart it if this overlap handoff fails.

The first 0.5 nm arbitrary-boundary step then produced a genuine local gain. The derivative predicted Δlog(Q) = +0.13146; the common-connectivity solve measured +0.11375 (86.5% of the prediction), raising surrogate Q from 1932.85 to 2165.70. Incumbent replay error was below 2×10⁻¹⁴, the candidate/incumbent field overlap was 0.99979, pole error was 4.73×10⁻⁵ µm⁻¹, localization was 0.71175, and the minimum feature remained 77.31 nm. The gradient moved both blocks: its RMS-normalized mixture weight was 0.976 for holes and 0.220 for the free cavity sidewall, which acquired 0.96 nm peak-to-peak modulation.

The physical seed inherits a nonzero deformation because a fresh mesh of the nearby checkpoint exceeded the host's 1.1-million-DOF safety ceiling. The old absolute element gate therefore rejected every candidate even when it made the mesh better. The corrected policy retains the nominal 0.35--3.0 element ratio bounds for an unstrained mesh, but grandfathers an overlap-qualified inherited extremum with at most 5% further deterioration. This first candidate actually improved both extrema: minimum element ratio 0.0887 → 0.1180 and maximum 3.751 → 3.630. All eleven pole, objective, geometry, overlap, transfer, and mesh checks then passed. The service resumed at 0.65 nm with zero restarts and no Tidy3D process or FlexCredit use.

After line-search contraction, a second accepted 0.08125 nm trust step raised surrogate Q from 2165.70 to 2200.08. The measured + 1.575% log-Q gain was 90.7% of the predicted + 1.737%, with mode overlap 0.999988. This is a frozen-connectivity surrogate result, not a new physical validation.

Left: the qualified Tidy3D seed is retained as the only physical point; the blue line is the separate FEM surrogate trajectory and must not be read as an FDTD prediction. The gray dotted marker at step 2 records the explicit motion-map rebaseline rather than a geometry gain. Right: complex-frequency motion from the known FEM reference to the exact qualified geometry on the same connectivity, selected by complex mass overlap. Sources: surrogate_seed_gate.json, state.json, optimizer_history.jsonl, and mesh_epoch_history.jsonl in run 20260813T133100Z_device23_surrogate_only_from_q1184; generator: plot_surrogate_only_handoff.py; exact values and axis scaling are in surrogate_only_handoff_manifest.json.

Matched-axis qualified seed and current accepted surrogate checkpoint, plus signed material change. Blue is Si₃N₄, white is air, red removes Si₃N₄, and green adds it. All 40 hole contours and the cavity-region sidewall are active; the sub-nanometre changes are intentionally subtle on a 10 µm view. Sources: checkpoints/field_step_000000.npz and the latest accepted checkpoint; native 4 nm level-set grid; generator and hashes in surrogate_only_handoff_manifest.json.

Removed boundary-motion floor

Attempts 6--8 all had positive measured surrogate-Q gains, including Q = 2221.36 at the minimum 0.05 nm trust amplitude, but the old boundary audit rejected them. That audit compared each incumbent marching-squares vertex with the nearest candidate vertex. A sub-grid contour perturbation changes where marching squares samples the same curve, so tangential vertex resampling produced a false 1.7--1.9 nm Hausdorff floor that did not contract with the requested step.

The optimizer now measures Hausdorff distance to the nearest continuous piecewise-linear contour segment. Normal boundary motion is obtained by solving for the zero of the stored candidate level-set field along each incumbent normal, with segment projection only as a recorded fallback. On the three blocked candidates, corrected Hausdorff distance is 0.296--0.655 nm and contracts with amplitude. For the old 0.05 nm trial, the corrected absolute ALE transfer error is 0.152043, below the unchanged 0.152956 trust limit; the corrected incumbent is 0.152161. Thus the blocked trial slightly improves rather than worsens this transfer metric.

Left: incremental contour Hausdorff distance for attempts 6--8 using the old nearest-vertex audit and the corrected segment/root audit. Right: old and corrected absolute ALE transfer errors for accepted step 2 and the formerly blocked 0.05 nm candidate; the red line is the unchanged production gate. Sources: batch_0000{6,7,8}.json, their candidate NPZ files, contraction_audit_incumbent.json, and contraction_audit_batch_00008.json; generator: plot_contraction_fix.py. Exact hashes, fallbacks, axes, and numerical values are in contraction_fix_manifest.json.

Changing the motion map also changes the ALE-deformed discrete operator, so the service performed one explicit incumbent rebaseline before promoting another step. This changed no canonical boundary: the same step-2 checkpoint replayed at Q = 2203.29 under the corrected map instead of 2200.08 under the old map, a 0.146% method offset.

The first corrected-map candidate then passed all eleven checks and raised surrogate Q from 2203.29 to 2227.44 at 752.637 nm. Its measured Δlog(Q) = +0.010899 slightly exceeded the predicted +0.010448; complex field overlap was 0.999988, pole tracking error was 1.39×10⁻⁵ µm⁻¹, minimum feature was 77.08 nm, and absolute transfer error was 0.152053 < 0.152956. The supervisor promoted this as accepted step 3, cleared the one-time rebaseline flag, and automatically started attempt 10 at 0.065 nm. Every later candidate is compared with an incumbent evaluated by the same map, and the service continues indefinitely until a STOP file is created or the service is stopped. Tidy3D remains disabled for this branch.

No new electromagnetic field is claimed for the unverified surrogate step. The live dashboard deliberately continues to show the retained qualified Tidy3D seed |E|² field at 768.555 nm in XY and XZ, with per-panel normalization and an explicit warning that it belongs to step 0.

Interpretation

The sidewall degree of freedom remains valuable. The failed run does not show that the cavity boundary lacks useful Q gradients; it shows that a dense raw boundary representation plus a non-invariant loss surrogate cannot expose them reliably. The next meaningful success criterion is not another increase of frozen-mesh Q. It is three consecutive qualified Tidy3D checkpoints with nonnegative change and at least 25% cumulative physical-Q improvement, followed by a 2× milestone before returning to the Q = 25,000 target.