Build a trustworthy validation case

Build a trustworthy validation case#

A benchmark should answer one narrow question and remain interpretable when it fails. This project uses a layered strategy.

1. State the physical claim#

Examples: “phase velocity in a dielectric is \(c/n\),” “the two-port straight guide transmits unit mode power,” or “the optimizer reaches 95% of the Tidy3D best FOM.” Name the observable before writing geometry.

2. Prefer independent references#

Use, in order:

  1. an analytical formula or conservation law;

  2. an independent numerical method such as transfer matrices or Mie theory;

  3. a frozen result from a converged external solver.

Tidy3D goldens in this repo are compact, pinned files. Normal benchmark execution never launches a paid task.

3. Declare resolution honestly#

If local GPU memory requires a coarser grid than the reference, set resolution_factor < 1 and declare resolution_power for every metric. The comparator uses

\[ \text{effective tolerance}=\text{base tolerance}\times (\text{resolution factor})^{-p}. \]

Document the reason in case.yaml; never quietly widen a threshold after looking at a failure.

4. Separate comparisons#

Record both shape and pointwise metrics for spectra (rel_l2, abs_max), plus conservation/finite checks where meaningful. A relative metric alone becomes unstable near zero; an absolute metric alone can miss scale-dependent error.

5. Treat inverse design differently#

Compare the best FOM reached within a generous cap. Do not require matching geometry or optimization speed. Add bounds that exclude unphysical normalization wins.

6. Run in the right order#

uv run fdtdx-bench run --case your_case
uv run fdtdx-bench smoke
uv run fdtdx-bench report

The smoke suite must remain green after solver changes. report refreshes the human-readable capability record from machine state.

7. Block rather than disguise#

After repeated sincere attempts, record a blocked reason. A non-converged golden, an ambiguous source convention, or a missing solver feature is valuable information when it is explicit. The public validation page includes those boundaries.