Benchmark protocol

Benchmark protocol#

The benchmark suite is designed to improve the solver without turning the test into the product.

Case anatomy#

Each benchmarks/cases/<id>/ directory contains:

  • case.yaml: identity, title, kind, resolution, tags, metric contract, notes, and optional golden path;

  • run.py: local setup, execution, reduction, and structured metrics.

benchmarks/catalog.yaml supplies ordering and global policy. progress.json records the latest completed result.

Reference hierarchy#

Analytical

Closed-form physics, conservation, reciprocity, or a separately implemented numerical method.

Forward parity

An FDTDX observable compared with a compact, frozen Tidy3D result. Goldens include provenance and hashes; cloud generation is outside the hot loop.

Inverse-design parity

Best objective reached locally compared with Tidy3D’s best objective. Geometry and wall-clock speed are not comparison targets.

Resolution-aware acceptance#

When a case uses a coarser grid than its golden, each metric declares a resolution_power \(p\) and uses

\[ a_\text{eff}=a_0r^{-p},\qquad t_\text{eff}=t_0r^{-p}, \]

where \(r\) is the linear resolution_factor, \(a\) is absolute tolerance, and \(t\) is relative tolerance. The case notes must explain the coarsening. This policy acknowledges discretization error without allowing ad-hoc post-failure tolerance edits.

Metric meanings#

Comparator

Question

scalar

Is one number within absolute + relative budget?

rel_l2

Does an array/spectrum have the right overall shape and scale?

abs_max

Is the worst pointwise error bounded?

corr

Is pattern correlation high enough?

fom_ge

Does best optimization performance meet or exceed a threshold?

Completed outputs must include exactly the declared finite metrics. Pass/fail is derived from them by the harness.

Inverse design policy#

The default criterion is

\[ J_{\text{best,FDTDX}}\geq J_{\text{best,Tidy3D}}\times\texttt{fom_ratio}, \]

within a generous cap of three times Tidy3D’s iteration count. Converging two or three times slower is acceptable. Additional passive-power and parameter-bound checks prevent objective loopholes.

Golden integrity#

  • goldens are never updated merely because a local case fails;

  • generators require an explicit cloud-run switch;

  • provenance links each artifact to builder, reducer, source, and hash;

  • non-converged or irreproducible references are quarantined as blocked;

  • tolerances and benchmark definitions remain reviewable YAML, not hidden runner constants.