# Build a trustworthy validation case

A benchmark should answer one narrow question and remain interpretable when it fails. This project uses a layered strategy.

## 1. State the physical claim

Examples: “phase velocity in a dielectric is $c/n$,” “the two-port straight guide transmits unit mode power,” or “the optimizer reaches 95% of the Tidy3D best FOM.” Name the observable before writing geometry.

## 2. Prefer independent references

Use, in order:

1. an analytical formula or conservation law;
2. an independent numerical method such as transfer matrices or Mie theory;
3. a frozen result from a converged external solver.

Tidy3D goldens in this repo are compact, pinned files. Normal benchmark execution never launches a paid task.

## 3. Declare resolution honestly

If local GPU memory requires a coarser grid than the reference, set `resolution_factor < 1` and declare `resolution_power` for every metric. The comparator uses

$$
\text{effective tolerance}=\text{base tolerance}\times
(\text{resolution factor})^{-p}.
$$

Document the reason in `case.yaml`; never quietly widen a threshold after looking at a failure.

## 4. Separate comparisons

Record both shape and pointwise metrics for spectra (`rel_l2`, `abs_max`), plus conservation/finite checks where meaningful. A relative metric alone becomes unstable near zero; an absolute metric alone can miss scale-dependent error.

## 5. Treat inverse design differently

Compare the best FOM reached within a generous cap. Do not require matching geometry or optimization speed. Add bounds that exclude unphysical normalization wins.

## 6. Run in the right order

```bash
uv run fdtdx-bench run --case your_case
uv run fdtdx-bench smoke
uv run fdtdx-bench report
```

The smoke suite must remain green after solver changes. `report` refreshes the human-readable capability record from machine state.

## 7. Block rather than disguise

After repeated sincere attempts, record a blocked reason. A non-converged golden, an ambiguous source convention, or a missing solver feature is valuable information when it is explicit. The public [validation page](../project/validation.md) includes those boundaries.